From 74ea045c38c7cd814d2e9e14f0200406f21ef2ed Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Mon, 13 Jul 2026 23:13:50 +0200 Subject: [PATCH 01/16] Add "Open folder as library" (directory as library, read-only scan) A directory can now be opened as a library: the main table fills from the Hayagriva .yml sidecars and PDFs found in the tree. Sidecar entries link their same-basename PDF relative to the root; PDFs without a sidecar become stub entries. The context gets the new DatabaseLocation.DIRECTORY with an empty database path, which keeps autosave/backup/change-monitor and path-dependent actions off by default; "Save as" converts the tab into a regular .bib library. GitIgnoreFileFilter moves to jablib for reuse by the scanner. Edits are not yet written back to the files; live synchronization and write-back follow in later steps (see PLAN.md on directory-as-library). Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 1 + ...tory-as-library-with-hayagriva-sidecars.md | 66 ++++++ docs/requirements/directory-library.md | 19 ++ .../main/java/org/jabref/gui/LibraryTab.java | 16 +- .../jabref/gui/actions/StandardActions.java | 1 + .../gui/exporter/SaveDatabaseAction.java | 3 + .../externalfiles/UnlinkedFilesCrawler.java | 1 + .../java/org/jabref/gui/frame/MainMenu.java | 2 + .../actions/OpenDirectoryLibraryAction.java | 102 +++++++++ jablib/src/main/java/module-info.java | 1 + .../DirectoryLibraryCatalog.java | 43 ++++ .../DirectoryLibraryScanner.java | 178 ++++++++++++++++ .../jabref/logic/shared/DatabaseLocation.java | 5 +- .../logic/util/io}/GitIgnoreFileFilter.java | 2 +- .../model/database/BibDatabaseContext.java | 15 ++ .../main/resources/l10n/JabRef_en.properties | 4 + .../DirectoryLibraryScannerTest.java | 196 ++++++++++++++++++ .../util/io}/GitIgnoreFileFilterTest.java | 2 +- 18 files changed, 652 insertions(+), 5 deletions(-) create mode 100644 docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md create mode 100644 docs/requirements/directory-library.md create mode 100644 jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java create mode 100644 jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryCatalog.java create mode 100644 jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java rename {jabgui/src/main/java/org/jabref/gui/externalfiles => jablib/src/main/java/org/jabref/logic/util/io}/GitIgnoreFileFilter.java (99%) create mode 100644 jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java rename {jabgui/src/test/java/org/jabref/gui/externalfiles => jablib/src/test/java/org/jabref/logic/util/io}/GitIgnoreFileFilterTest.java (98%) diff --git a/CHANGELOG.md b/CHANGELOG.md index 52aa746afc36..03523016fdd9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv ### Added +- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear as stub entries titled after the file. Edits are not yet written back to the files. [TODO] - The HTTP import endpoint (`POST /libraries/{id}/entries`) now accepts CSL-JSON (`application/vnd.citationstyles.csl+json`), mapping each item to the correct entry type (e.g. conference paper, book chapter, thesis) via the citation-js-based mapping. [#16151](https://github.com/JabRef/jabref/pull/16151) - We added a new "Main" tab to the entry editor showing all fields of an entry in a single scrollable list, with one-click chips for adding optional fields and a free-form box for adding arbitrary fields. Identifiers, files and links, bibliometrics, comments, and meta fields (groups, owner, timestamps, special fields) live in collapsible sections — collapsed when empty — each offering chips for its unset fields. [#12711](https://github.com/JabRef/jabref/issues/12711) - We added auto-detection import for drag-and-dropped library files. [#15391](https://github.com/JabRef/jabref/issues/15391) diff --git a/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md b/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md new file mode 100644 index 000000000000..edd9566c4b30 --- /dev/null +++ b/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md @@ -0,0 +1,66 @@ +--- +nav_order: 0066 +parent: Decision Records +--- + +# Directory as library with Hayagriva sidecars + +## Context and Problem Statement + +Users organize papers as PDFs in folder trees (often synced via cloud storage) and want JabRef +to treat such a folder directly as a library, without maintaining a separate `.bib` file. Each +work's bibliographic data (including notes) should live in a plain-text file next to its PDF, so +the folder stays usable from other tools (e.g. Typst). How should such a "directory library" be +represented in JabRef, and where does the per-entry data live? + +## Decision Drivers + +* The folder must remain self-describing and usable outside JabRef. +* JabRef's existing architecture assumes a `BibDatabaseContext` per library tab. +* File changes must eventually sync in both directions (external edits appear live, JabRef edits + persist back) without echo loops. +* The groups panel should later mirror the folder structure ([#10930](https://github.com/JabRef/jabref/issues/10930)). + +## Considered Options + +* Hayagriva YAML sidecars (`X.yml` next to `X.pdf`), directory represented as a third + `DatabaseLocation` with an empty database path +* A hidden auto-maintained `.bib` file inside the directory +* XMP metadata embedded in the PDFs as the only store + +## Decision Outcome + +Chosen option: "Hayagriva YAML sidecars with a third `DatabaseLocation`", because sidecars keep +the folder tool-agnostic (Hayagriva is Typst's bibliography format and JabRef has a symmetric +importer/exporter for it, including the `note` field), embedded XMP cannot represent all fields +and rewrites the PDFs themselves, and a hidden `.bib` would duplicate state that immediately +drifts from the files. + +Key points of the chosen design: + +* `DatabaseLocation.DIRECTORY`: the context keeps an **empty** database path plus a separate + directory root. Empty path gives correct default behavior at almost every existing decision + point (no autosave/backup managers, no `.bib` change monitor, "needs saved local database" + actions disabled). The directory root is registered as the library-specific file directory, so + relative PDF links resolve without a database path. +* Pairing is by convention — `X.yml`/`X.yaml` next to `X.pdf` — because Hayagriva has no + file-path field; nothing JabRef-specific is written into the YAML for the association. +* Non-Hayagriva `.yml` files (CI configs, ...) are ignored via format recognition, not reported + as errors. PDFs without a sidecar become stub entries; a sidecar is only written once the user + edits the entry (scanning never writes files). +* Later synchronization mirrors the shared-SQL seam (`convertToSharedDatabase` / + `DBMSSynchronizer`): a directory synchronizer subscribes to entry events through a + `CoarseChangeFilter` for write-back and applies inbound file changes with a non-local + `EntriesEventSource` to prevent echo loops; directory watching uses the Commons-IO + `DirectoryMonitor` ([ADR-0030](0030-use-apache-commons-io-for-directory-monitoring.md)). + "Save as" converts a directory library into a regular `.bib` library. + +### Consequences + +* Good, because the directory stays the single source of truth and is usable from Typst as-is. +* Good, because the empty-database-path representation needs only a handful of explicit UI + branches (tab title, close confirmation, save-as). +* Bad, because YAML comments in hand-edited sidecars will not survive JabRef rewrites (the YAML + parser drops them), and JabRef-only fields need an extension mechanism inside the entry. +* Bad, because library-level metadata (groups, save actions) has no natural home yet; a + metadata file in the root may be added later. diff --git a/docs/requirements/directory-library.md b/docs/requirements/directory-library.md new file mode 100644 index 000000000000..38e60f7897dd --- /dev/null +++ b/docs/requirements/directory-library.md @@ -0,0 +1,19 @@ +--- +parent: Requirements +--- +# Directory library + +## Directory scan builds the library from sidecars and PDFs +`req~directory-library.scan~1` + +Opening a directory as a library must fill the library from the directory tree: every Hayagriva +`.yml`/`.yaml` file contributes its entries, a PDF with the same base name next to a sidecar is +linked to the sidecar's entry, and PDFs without a sidecar become stub entries titled after the +file. Hidden files/directories, gitignored paths, and `.yml` files not recognized as Hayagriva +are skipped; unparseable Hayagriva files are reported as warnings without aborting the scan. +Scanning must not write or modify any file in the directory. +See [ADR 66](../decisions/0066-directory-as-library-with-hayagriva-sidecars.md) for more details. + +Needs: impl + + diff --git a/jabgui/src/main/java/org/jabref/gui/LibraryTab.java b/jabgui/src/main/java/org/jabref/gui/LibraryTab.java index b835e4d1baef..84a8ad03cd17 100644 --- a/jabgui/src/main/java/org/jabref/gui/LibraryTab.java +++ b/jabgui/src/main/java/org/jabref/gui/LibraryTab.java @@ -472,12 +472,21 @@ public void updateTabTitle(boolean isChanged) { } else { tabTitle.append(Localization.lang("untitled")); } + } else if (databaseLocation == DatabaseLocation.DIRECTORY) { + if (isChanged) { + tabTitle.append('*'); + } + bibDatabaseContext.getDirectoryLibraryRoot().ifPresent(root -> { + tabTitle.append(root.getFileName().toString()); + toolTipText.append(root.toAbsolutePath()); + }); } else { addSharedDbInformation(tabTitle, bibDatabaseContext); addSharedDbInformation(toolTipText, bibDatabaseContext); } addModeInfo(toolTipText, bibDatabaseContext); - if ((databaseLocation == DatabaseLocation.LOCAL) && bibDatabaseContext.getDatabase().hasEntries()) { + if ((databaseLocation == DatabaseLocation.LOCAL || databaseLocation == DatabaseLocation.DIRECTORY) + && bibDatabaseContext.getDatabase().hasEntries()) { addChangedInformation(toolTipText); } } @@ -647,7 +656,9 @@ private boolean showDeleteConfirmationDialog(int numberOfEntries) { } public boolean requestClose() { - if (bibDatabaseContext.getLocation() == DatabaseLocation.LOCAL) { + // DIRECTORY prompts as well: until file write-back exists, edits are in-memory only + if (bibDatabaseContext.getLocation() == DatabaseLocation.LOCAL + || bibDatabaseContext.getLocation() == DatabaseLocation.DIRECTORY) { if (isModified()) { return confirmClose(); } @@ -669,6 +680,7 @@ private boolean confirmClose() { String filename = getBibDatabaseContext() .getDatabasePath() + .or(() -> getBibDatabaseContext().getDirectoryLibraryRoot()) .map(Path::toAbsolutePath) .map(Path::toString) .orElse(Localization.lang("untitled")); diff --git a/jabgui/src/main/java/org/jabref/gui/actions/StandardActions.java b/jabgui/src/main/java/org/jabref/gui/actions/StandardActions.java index 6ee78462022d..28403c55fc3f 100644 --- a/jabgui/src/main/java/org/jabref/gui/actions/StandardActions.java +++ b/jabgui/src/main/java/org/jabref/gui/actions/StandardActions.java @@ -78,6 +78,7 @@ public enum StandardActions implements Action { RELEVANT(Localization.lang("Toggle relevance"), IconTheme.JabRefIcons.RELEVANCE), NEW_LIBRARY(Localization.lang("New empty library"), IconTheme.JabRefIcons.NEW), OPEN_LIBRARY(Localization.lang("Open library..."), IconTheme.JabRefIcons.OPEN, KeyBinding.OPEN_LIBRARY), + OPEN_FOLDER_AS_LIBRARY(Localization.lang("Open folder as library..."), IconTheme.JabRefIcons.OPEN), MERGE_LIBRARY(Localization.lang("Merge..."), IconTheme.JabRefIcons.MERGE_ENTRIES), IMPORT(Localization.lang("Import"), IconTheme.JabRefIcons.IMPORT), EXPORT(Localization.lang("Export"), IconTheme.JabRefIcons.EXPORT, KeyBinding.EXPORT), diff --git a/jabgui/src/main/java/org/jabref/gui/exporter/SaveDatabaseAction.java b/jabgui/src/main/java/org/jabref/gui/exporter/SaveDatabaseAction.java index 696685aa024c..246a66fc11d3 100644 --- a/jabgui/src/main/java/org/jabref/gui/exporter/SaveDatabaseAction.java +++ b/jabgui/src/main/java/org/jabref/gui/exporter/SaveDatabaseAction.java @@ -145,6 +145,9 @@ boolean saveAs(Path file, SaveDatabaseMode mode) { // Save all properties dependent on the ID. This makes it possible to restore them. new SharedDatabasePreferences(context.getDatabase().generateSharedDatabaseID()) .putAllDBMSConnectionProperties(context.getDBMSSynchronizer().getConnectionProperties()); + } else if (context.getLocation() == DatabaseLocation.DIRECTORY) { + // "Save as" snapshots a directory library into a regular .bib library + context.convertToLocalDatabase(); } boolean saveResult = save(file, mode); diff --git a/jabgui/src/main/java/org/jabref/gui/externalfiles/UnlinkedFilesCrawler.java b/jabgui/src/main/java/org/jabref/gui/externalfiles/UnlinkedFilesCrawler.java index 754e7be4ef81..cce66274134c 100644 --- a/jabgui/src/main/java/org/jabref/gui/externalfiles/UnlinkedFilesCrawler.java +++ b/jabgui/src/main/java/org/jabref/gui/externalfiles/UnlinkedFilesCrawler.java @@ -17,6 +17,7 @@ import org.jabref.logic.externalfiles.DateRange; import org.jabref.logic.externalfiles.ExternalFileSorter; import org.jabref.logic.util.BackgroundTask; +import org.jabref.logic.util.io.GitIgnoreFileFilter; import org.jabref.model.database.BibDatabaseContext; import org.slf4j.Logger; diff --git a/jabgui/src/main/java/org/jabref/gui/frame/MainMenu.java b/jabgui/src/main/java/org/jabref/gui/frame/MainMenu.java index 0f34b624267a..0d70fddb6772 100644 --- a/jabgui/src/main/java/org/jabref/gui/frame/MainMenu.java +++ b/jabgui/src/main/java/org/jabref/gui/frame/MainMenu.java @@ -50,6 +50,7 @@ import org.jabref.gui.importer.NewEntryAction; import org.jabref.gui.importer.actions.ImportCommand; import org.jabref.gui.importer.actions.OpenDatabaseAction; +import org.jabref.gui.importer.actions.OpenDirectoryLibraryAction; import org.jabref.gui.importer.fetcher.LookupIdentifierAction; import org.jabref.gui.integrity.IntegrityCheckAction; import org.jabref.gui.libraryproperties.LibraryPropertiesAction; @@ -173,6 +174,7 @@ private void createMenu() { file.getItems().addAll( factory.createMenuItem(StandardActions.NEW_LIBRARY, new NewDatabaseAction(frame, preferences)), factory.createMenuItem(StandardActions.OPEN_LIBRARY, openDatabaseActionSupplier.get()), + factory.createMenuItem(StandardActions.OPEN_FOLDER_AS_LIBRARY, new OpenDirectoryLibraryAction(frame, dialogService, preferences, aiService, stateManager, fileUpdateMonitor, entryTypesManager, undoManager, clipBoardManager, taskExecutor)), fileHistoryMenu, factory.createMenuItem(StandardActions.SAVE_LIBRARY, new SaveAction(SaveAction.SaveMethod.SAVE, frame::getCurrentLibraryTab, dialogService, preferences, stateManager)), factory.createMenuItem(StandardActions.SAVE_LIBRARY_AS, new SaveAction(SaveAction.SaveMethod.SAVE_AS, frame::getCurrentLibraryTab, dialogService, preferences, stateManager)), diff --git a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java new file mode 100644 index 000000000000..5bb036cfbc79 --- /dev/null +++ b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java @@ -0,0 +1,102 @@ +package org.jabref.gui.importer.actions; + +import java.nio.file.Path; + +import javax.swing.undo.UndoManager; + +import org.jabref.gui.DialogService; +import org.jabref.gui.LibraryTab; +import org.jabref.gui.LibraryTabContainer; +import org.jabref.gui.StateManager; +import org.jabref.gui.actions.SimpleCommand; +import org.jabref.gui.clipboard.ClipBoardManager; +import org.jabref.gui.preferences.GuiPreferences; +import org.jabref.gui.util.DirectoryDialogConfiguration; +import org.jabref.logic.ai.AiService; +import org.jabref.logic.directorylibrary.DirectoryLibraryScanner; +import org.jabref.logic.l10n.Localization; +import org.jabref.logic.util.BackgroundTask; +import org.jabref.logic.util.TaskExecutor; +import org.jabref.model.entry.BibEntryTypesManager; +import org.jabref.model.util.FileUpdateMonitor; + +/// Opens a directory as a library: the main table fills from the Hayagriva `.yml` sidecars and +/// `.pdf` files found in the directory tree (see [DirectoryLibraryScanner]). +public class OpenDirectoryLibraryAction extends SimpleCommand { + + private final LibraryTabContainer tabContainer; + private final DialogService dialogService; + private final GuiPreferences preferences; + private final AiService aiService; + private final StateManager stateManager; + private final FileUpdateMonitor fileUpdateMonitor; + private final BibEntryTypesManager entryTypesManager; + private final UndoManager undoManager; + private final ClipBoardManager clipBoardManager; + private final TaskExecutor taskExecutor; + + public OpenDirectoryLibraryAction(LibraryTabContainer tabContainer, + DialogService dialogService, + GuiPreferences preferences, + AiService aiService, + StateManager stateManager, + FileUpdateMonitor fileUpdateMonitor, + BibEntryTypesManager entryTypesManager, + UndoManager undoManager, + ClipBoardManager clipBoardManager, + TaskExecutor taskExecutor) { + this.tabContainer = tabContainer; + this.dialogService = dialogService; + this.preferences = preferences; + this.aiService = aiService; + this.stateManager = stateManager; + this.fileUpdateMonitor = fileUpdateMonitor; + this.entryTypesManager = entryTypesManager; + this.undoManager = undoManager; + this.clipBoardManager = clipBoardManager; + this.taskExecutor = taskExecutor; + } + + @Override + public void execute() { + DirectoryDialogConfiguration directoryDialogConfiguration = new DirectoryDialogConfiguration.Builder() + .withInitialDirectory(preferences.getFilePreferences().getWorkingDirectory()) + .build(); + dialogService.showDirectorySelectionDialog(directoryDialogConfiguration) + .ifPresent(this::openDirectory); + } + + private void openDirectory(Path root) { + preferences.getFilePreferences().setWorkingDirectory(root); + BackgroundTask.wrap(() -> new DirectoryLibraryScanner().scan(root)) + .onSuccess(this::showLibraryTab) + .onFailure(exception -> dialogService.showErrorDialogAndWait( + Localization.lang("Open folder as library"), + Localization.lang("Could not open folder '%0' as library.", root.toString()), + exception)) + .executeWith(taskExecutor); + } + + private void showLibraryTab(DirectoryLibraryScanner.ScanResult scanResult) { + // The synchronous factory keeps the DIRECTORY location: the ParserResult-based one + // reconstructs a fresh (LOCAL) context from database + metadata on loading success + LibraryTab libraryTab = LibraryTab.createLibraryTab( + scanResult.databaseContext(), + tabContainer, + dialogService, + aiService, + preferences, + stateManager, + fileUpdateMonitor, + entryTypesManager, + undoManager, + clipBoardManager, + taskExecutor); + tabContainer.addTab(libraryTab, true); + if (!scanResult.warnings().isEmpty()) { + dialogService.showWarningDialogAndWait( + Localization.lang("Open folder as library"), + String.join("\n", scanResult.warnings())); + } + } +} diff --git a/jablib/src/main/java/module-info.java b/jablib/src/main/java/module-info.java index e8683caefb8b..6ffac1fbae59 100644 --- a/jablib/src/main/java/module-info.java +++ b/jablib/src/main/java/module-info.java @@ -101,6 +101,7 @@ exports org.jabref.logic.shared.event; exports org.jabref.logic.citation; exports org.jabref.logic.crawler; + exports org.jabref.logic.directorylibrary; exports org.jabref.logic.pseudonymization; exports org.jabref.logic.citation.repository; exports org.jabref.model.paging; diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryCatalog.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryCatalog.java new file mode 100644 index 000000000000..76b38d469522 --- /dev/null +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryCatalog.java @@ -0,0 +1,43 @@ +package org.jabref.logic.directorylibrary; + +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +import org.jabref.model.entry.BibEntry; + +import org.jspecify.annotations.NullMarked; + +/// Remembers which file of a directory library each [BibEntry] came from. This is the identity +/// backbone of a directory library: entries are keyed by their stable in-memory id +/// ([BibEntry#getId()]), so citation-key edits do not lose the association, and the Hayagriva +/// key (the top-level YAML map key) is tracked separately so it can be renamed on write-back. +/// +/// Entries without a source (e.g. stubs created for PDFs that have no sidecar yet) are simply +/// absent; they get registered once a sidecar is written for them. +@NullMarked +public class DirectoryLibraryCatalog { + + public record EntrySource(Path yamlFile, String hayagrivaKey) { + } + + private final Map sourceByEntryId = new HashMap<>(); + private final Map> entryIdsByFile = new HashMap<>(); + + public void register(BibEntry entry, Path yamlFile, String hayagrivaKey) { + sourceByEntryId.put(entry.getId(), new EntrySource(yamlFile, hayagrivaKey)); + entryIdsByFile.computeIfAbsent(yamlFile, _ -> new ArrayList<>()).add(entry.getId()); + } + + public Optional sourceOf(BibEntry entry) { + return Optional.ofNullable(sourceByEntryId.get(entry.getId())); + } + + /// Entry ids of all entries read from the given file, in file order. + public List entryIdsIn(Path yamlFile) { + return List.copyOf(entryIdsByFile.getOrDefault(yamlFile, List.of())); + } +} diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java new file mode 100644 index 000000000000..cc9f91d99900 --- /dev/null +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java @@ -0,0 +1,178 @@ +package org.jabref.logic.directorylibrary; + +import java.io.BufferedReader; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.FileVisitResult; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.SimpleFileVisitor; +import java.nio.file.attribute.BasicFileAttributes; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Optional; +import java.util.Set; + +import org.jabref.logic.importer.ParserResult; +import org.jabref.logic.importer.fileformat.HayagrivaImporter; +import org.jabref.logic.l10n.Localization; +import org.jabref.logic.util.StandardFileType; +import org.jabref.logic.util.io.FileUtil; +import org.jabref.logic.util.io.GitIgnoreFileFilter; +import org.jabref.model.database.BibDatabaseContext; +import org.jabref.model.entry.BibEntry; +import org.jabref.model.entry.LinkedFile; +import org.jabref.model.entry.field.StandardField; +import org.jabref.model.entry.types.StandardEntryType; + +import org.jspecify.annotations.NullMarked; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/// Builds an in-memory library from a directory tree: each Hayagriva `.yml`/`.yaml` file +/// contributes its entries, a PDF next to a sidecar of the same base name is linked to the +/// sidecar's (first) entry, and PDFs without a sidecar become stub entries titled after the +/// file. The directory itself is the library — the resulting [BibDatabaseContext] has +/// [org.jabref.logic.shared.DatabaseLocation#DIRECTORY] and no database path; linked files are +/// stored relative to the root, which is registered as the library-specific file directory. +// [impl->req~directory-library.scan~1] +@NullMarked +public class DirectoryLibraryScanner { + + /// Everything a directory scan produces: the ready-to-open context, the entry-to-file + /// catalog (consumed by the file synchronization in later steps), and user-facing warnings + /// about files that looked like Hayagriva but could not be parsed. + public record ScanResult(BibDatabaseContext databaseContext, DirectoryLibraryCatalog catalog, List warnings) { + } + + private static final Logger LOGGER = LoggerFactory.getLogger(DirectoryLibraryScanner.class); + + private static final Set YAML_EXTENSIONS = Set.of("yml", "yaml"); + private static final String PDF_EXTENSION = "pdf"; + + private final HayagrivaImporter importer = new HayagrivaImporter(); + + public ScanResult scan(Path root) throws IOException { + BibDatabaseContext databaseContext = new BibDatabaseContext(); + databaseContext.convertToDirectoryLibrary(root); + // Makes the relative PDF links resolvable although the context has no database path + databaseContext.getMetaData().setLibrarySpecificFileDirectory(root.toAbsolutePath().toString()); + + DirectoryLibraryCatalog catalog = new DirectoryLibraryCatalog(); + List warnings = new ArrayList<>(); + + List yamlFiles = new ArrayList<>(); + List pdfFiles = new ArrayList<>(); + collectFiles(root, yamlFiles, pdfFiles); + + List entries = new ArrayList<>(); + Set pairedPdfs = new HashSet<>(); + for (Path yamlFile : yamlFiles) { + // A directory may contain arbitrary YAML (CI configs, ...) — only files recognized + // as Hayagriva become entries; others are silently ignored + if (!looksLikeHayagriva(yamlFile)) { + continue; + } + ParserResult parserResult = importer.importDatabase(yamlFile); + List fileEntries = parserResult.getDatabase().getEntries(); + if (parserResult.isInvalid() || fileEntries.isEmpty()) { + warnings.add(Localization.lang("Could not parse the Hayagriva file '%0'.", yamlFile.toString())); + continue; + } + fileEntries.forEach(entry -> catalog.register(entry, yamlFile, entry.getCitationKey().orElse(""))); + findPairedPdf(yamlFile).ifPresent(pdf -> { + pairedPdfs.add(pdf); + linkPdf(fileEntries.getFirst(), root, pdf); + }); + entries.addAll(fileEntries); + } + + for (Path pdf : pdfFiles) { + if (pairedPdfs.contains(pdf)) { + continue; + } + // No metadata extraction here: the stub only makes the PDF visible; a sidecar is + // written once the user edits the entry + BibEntry stub = new BibEntry(StandardEntryType.Misc) + .withField(StandardField.TITLE, FileUtil.getBaseName(pdf)); + linkPdf(stub, root, pdf); + entries.add(stub); + } + + databaseContext.getDatabase().insertEntries(entries); + return new ScanResult(databaseContext, catalog, warnings); + } + + private void collectFiles(Path root, List yamlFiles, List pdfFiles) throws IOException { + GitIgnoreFileFilter gitIgnoreFilter = new GitIgnoreFileFilter(root); + Files.walkFileTree(root, new SimpleFileVisitor<>() { + @Override + public FileVisitResult preVisitDirectory(Path directory, BasicFileAttributes attributes) throws IOException { + if (!directory.equals(root) && (isHidden(directory) || !gitIgnoreFilter.accept(directory))) { + return FileVisitResult.SKIP_SUBTREE; + } + return FileVisitResult.CONTINUE; + } + + @Override + public FileVisitResult visitFile(Path file, BasicFileAttributes attributes) throws IOException { + if (isHidden(file) || !gitIgnoreFilter.accept(file)) { + return FileVisitResult.CONTINUE; + } + String extension = FileUtil.getFileExtension(file).orElse(""); + if (YAML_EXTENSIONS.contains(extension)) { + yamlFiles.add(file); + } else if (PDF_EXTENSION.equals(extension)) { + pdfFiles.add(file); + } + return FileVisitResult.CONTINUE; + } + + @Override + public FileVisitResult visitFileFailed(Path file, IOException exception) { + LOGGER.debug("Skipping unreadable path {}", file, exception); + return FileVisitResult.CONTINUE; + } + }); + yamlFiles.sort(Path::compareTo); + pdfFiles.sort(Path::compareTo); + } + + private static boolean isHidden(Path path) { + Path fileName = path.getFileName(); + return fileName != null && fileName.toString().startsWith("."); + } + + /// Uses the lookahead-based recognition (a `type:` line naming a Hayagriva entry type) + /// instead of [org.jabref.logic.importer.Importer#isRecognizedFormat(Path)], which fully + /// parses the YAML: a syntactically broken sidecar must surface as a warning, not be + /// silently skipped as "not Hayagriva". + private boolean looksLikeHayagriva(Path yamlFile) throws IOException { + try (BufferedReader reader = Files.newBufferedReader(yamlFile, StandardCharsets.UTF_8)) { + return importer.isRecognizedFormat(reader); + } + } + + /// The sidecar convention: `X.yml` next to `X.pdf` (same directory, same base name). + private Optional findPairedPdf(Path yamlFile) { + Path parent = yamlFile.getParent(); + if (parent == null) { + return Optional.empty(); + } + String baseName = FileUtil.getBaseName(yamlFile); + for (String candidate : List.of(baseName + ".pdf", baseName + ".PDF")) { + Path pdf = parent.resolve(candidate); + if (Files.exists(pdf)) { + return Optional.of(pdf); + } + } + return Optional.empty(); + } + + private void linkPdf(BibEntry entry, Path root, Path pdf) { + if (entry.getFiles().isEmpty()) { + entry.addFile(new LinkedFile("", root.relativize(pdf), StandardFileType.PDF.getName())); + } + } +} diff --git a/jablib/src/main/java/org/jabref/logic/shared/DatabaseLocation.java b/jablib/src/main/java/org/jabref/logic/shared/DatabaseLocation.java index 318629be64b7..8593e6008d7c 100644 --- a/jablib/src/main/java/org/jabref/logic/shared/DatabaseLocation.java +++ b/jablib/src/main/java/org/jabref/logic/shared/DatabaseLocation.java @@ -3,5 +3,8 @@ /// This enum represents the location for {@link org.jabref.model.database.BibDatabaseContext}. public enum DatabaseLocation { LOCAL, - SHARED + SHARED, + /// A directory opened as a library: entries live in Hayagriva YAML files next to their PDFs, + /// there is no `.bib` file, and thus no database path. + DIRECTORY } diff --git a/jabgui/src/main/java/org/jabref/gui/externalfiles/GitIgnoreFileFilter.java b/jablib/src/main/java/org/jabref/logic/util/io/GitIgnoreFileFilter.java similarity index 99% rename from jabgui/src/main/java/org/jabref/gui/externalfiles/GitIgnoreFileFilter.java rename to jablib/src/main/java/org/jabref/logic/util/io/GitIgnoreFileFilter.java index cca10320b7c0..1e628fa043ab 100644 --- a/jabgui/src/main/java/org/jabref/gui/externalfiles/GitIgnoreFileFilter.java +++ b/jablib/src/main/java/org/jabref/logic/util/io/GitIgnoreFileFilter.java @@ -1,4 +1,4 @@ -package org.jabref.gui.externalfiles; +package org.jabref.logic.util.io; import java.io.IOException; import java.nio.file.DirectoryStream; diff --git a/jablib/src/main/java/org/jabref/model/database/BibDatabaseContext.java b/jablib/src/main/java/org/jabref/model/database/BibDatabaseContext.java index dae1f6cdbcb6..648050f132ea 100644 --- a/jablib/src/main/java/org/jabref/model/database/BibDatabaseContext.java +++ b/jablib/src/main/java/org/jabref/model/database/BibDatabaseContext.java @@ -68,6 +68,11 @@ public class BibDatabaseContext { @Nullable private CoarseChangeFilter dbmsListener; + /// The root of a [DatabaseLocation#DIRECTORY] library; unrelated to [#path], which stays + /// empty for directory libraries. + @Nullable + private Path directoryLibraryRoot; + private DatabaseLocation location; public BibDatabaseContext() { @@ -269,6 +274,15 @@ public void convertToSharedDatabase(DatabaseSynchronizer dbmsSynchronizer) { this.location = DatabaseLocation.SHARED; } + public void convertToDirectoryLibrary(Path root) { + this.directoryLibraryRoot = root; + this.location = DatabaseLocation.DIRECTORY; + } + + public Optional getDirectoryLibraryRoot() { + return Optional.ofNullable(directoryLibraryRoot); + } + public void convertToLocalDatabase() { if (dbmsListener != null && (location == DatabaseLocation.SHARED)) { if (dbmsSynchronizer != null) { @@ -277,6 +291,7 @@ public void convertToLocalDatabase() { dbmsListener.shutdown(); } + this.directoryLibraryRoot = null; this.location = DatabaseLocation.LOCAL; } diff --git a/jablib/src/main/resources/l10n/JabRef_en.properties b/jablib/src/main/resources/l10n/JabRef_en.properties index 6c0a49d618e1..9abfc8c63c66 100644 --- a/jablib/src/main/resources/l10n/JabRef_en.properties +++ b/jablib/src/main/resources/l10n/JabRef_en.properties @@ -763,6 +763,8 @@ Select\ entry\ to\ link=Select entry to link new=new Open\ file=Open file +Open\ folder\ as\ library=Open folder as library +Open\ folder\ as\ library...=Open folder as library... Error\ opening\ file=Error opening file Error\ opening\ file\ '%0'=Error opening file '%0' Open\ file\ explorer=Open file explorer @@ -1216,6 +1218,8 @@ Choose\ OpenOffice/LibreOffice\ executable=Choose OpenOffice/LibreOffice executa Select\ document=Select document HTML\ list=HTML list Could\ not\ open\ %0=Could not open %0 +Could\ not\ open\ folder\ '%0'\ as\ library.=Could not open folder '%0' as library. +Could\ not\ parse\ the\ Hayagriva\ file\ '%0'.=Could not parse the Hayagriva file '%0'. Unknown\ import\ format=Unknown import format Could\ not\ auto-detect\ file\ format.\ An\ empty\ entry\ was\ created\ with\ file\ link.=Could not auto-detect file format. An empty entry was created with file link. No\ importable\ data\ was\ found\ in\ %0.\ An\ empty\ entry\ was\ created\ with\ file\ link.=No importable data was found in %0. An empty entry was created with file link. diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java new file mode 100644 index 000000000000..abc320e348c0 --- /dev/null +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java @@ -0,0 +1,196 @@ +package org.jabref.logic.directorylibrary; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.Optional; + +import org.jabref.logic.directorylibrary.DirectoryLibraryScanner.ScanResult; +import org.jabref.logic.shared.DatabaseLocation; +import org.jabref.model.entry.BibEntry; +import org.jabref.model.entry.LinkedFile; +import org.jabref.model.entry.field.StandardField; +import org.jabref.model.entry.types.StandardEntryType; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class DirectoryLibraryScannerTest { + + private static final String ARTICLE_YAML = """ + smith2020: + type: article + title: A Test Article + author: Smith, Jane + date: 2020-10-14 + note: Read twice + """; + + @TempDir + Path root; + + private final DirectoryLibraryScanner scanner = new DirectoryLibraryScanner(); + + private ScanResult scan() throws IOException { + return scanner.scan(root); + } + + private BibEntry singleEntry(ScanResult result) { + List entries = result.databaseContext().getDatabase().getEntries(); + assertEquals(1, entries.size()); + return entries.getFirst(); + } + + @Test + void contextIsDirectoryLibraryRootedAtTheScannedDirectory() throws IOException { + ScanResult result = scan(); + + assertEquals(DatabaseLocation.DIRECTORY, result.databaseContext().getLocation()); + assertEquals(Optional.of(root), result.databaseContext().getDirectoryLibraryRoot()); + assertEquals(Optional.of(root.toAbsolutePath().toString()), + result.databaseContext().getMetaData().getLibrarySpecificFileDirectory()); + assertEquals(Optional.empty(), result.databaseContext().getDatabasePath()); + } + + @Test + void sidecarEntryIsImportedAndPairedWithPdf() throws IOException { + Files.writeString(root.resolve("smith2020.yml"), ARTICLE_YAML); + Files.createFile(root.resolve("smith2020.pdf")); + + ScanResult result = scan(); + BibEntry entry = singleEntry(result); + + assertEquals(Optional.of("smith2020"), entry.getCitationKey()); + assertEquals(Optional.of("Read twice"), entry.getField(StandardField.NOTE)); + assertEquals(List.of(new LinkedFile("", Path.of("smith2020.pdf"), "PDF")), entry.getFiles()); + assertEquals(Optional.of(new DirectoryLibraryCatalog.EntrySource(root.resolve("smith2020.yml"), "smith2020")), + result.catalog().sourceOf(entry)); + } + + @Test + void sidecarInNestedDirectoryLinksPdfRelativeToRoot() throws IOException { + Path subDirectory = Files.createDirectories(root.resolve("conference").resolve("2020")); + Files.writeString(subDirectory.resolve("smith2020.yaml"), ARTICLE_YAML); + Files.createFile(subDirectory.resolve("smith2020.pdf")); + + BibEntry entry = singleEntry(scan()); + + assertEquals(List.of(new LinkedFile("", Path.of("conference", "2020", "smith2020.pdf"), "PDF")), + entry.getFiles()); + } + + @Test + void bareYamlYieldsEntryWithoutFileLink() throws IOException { + Files.writeString(root.resolve("smith2020.yml"), ARTICLE_YAML); + + BibEntry entry = singleEntry(scan()); + + assertEquals(Optional.of("smith2020"), entry.getCitationKey()); + assertEquals(List.of(), entry.getFiles()); + } + + @Test + void barePdfBecomesStubEntryTitledAfterTheFile() throws IOException { + Files.createFile(root.resolve("interesting-paper.pdf")); + + ScanResult result = scan(); + BibEntry entry = singleEntry(result); + + assertEquals(StandardEntryType.Misc, entry.getType()); + assertEquals(Optional.of("interesting-paper"), entry.getField(StandardField.TITLE)); + assertEquals(List.of(new LinkedFile("", Path.of("interesting-paper.pdf"), "PDF")), entry.getFiles()); + assertEquals(Optional.empty(), result.catalog().sourceOf(entry)); + } + + @Test + void multiEntryFileYieldsAllEntriesRegisteredToTheSameFile() throws IOException { + Files.writeString(root.resolve("collection.yml"), """ + first: + type: article + title: First + second: + type: book + title: Second + """); + + ScanResult result = scan(); + + List entries = result.databaseContext().getDatabase().getEntries(); + assertEquals(2, entries.size()); + assertEquals(2, result.catalog().entryIdsIn(root.resolve("collection.yml")).size()); + } + + @Test + void nonHayagrivaYamlIsIgnoredSilently() throws IOException { + Files.writeString(root.resolve("ci-config.yml"), """ + jobs: + build: + runs-on: ubuntu-latest + """); + + ScanResult result = scan(); + + assertEquals(List.of(), result.databaseContext().getDatabase().getEntries()); + assertEquals(List.of(), result.warnings()); + } + + @Test + void unparseableHayagrivaFileIsReportedAsWarning() throws IOException { + Files.writeString(root.resolve("broken.yml"), """ + broken: + type: article + title: [unclosed + """); + + ScanResult result = scan(); + + assertEquals(List.of(), result.databaseContext().getDatabase().getEntries()); + assertEquals(1, result.warnings().size()); + assertTrue(result.warnings().getFirst().contains("broken.yml")); + } + + @Test + void hiddenAndGitDirectoriesAreSkipped() throws IOException { + Path hidden = Files.createDirectories(root.resolve(".git")); + Files.writeString(hidden.resolve("smith2020.yml"), ARTICLE_YAML); + Files.writeString(root.resolve(".hidden.yml"), ARTICLE_YAML); + + ScanResult result = scan(); + + assertEquals(List.of(), result.databaseContext().getDatabase().getEntries()); + } + + @Test + void gitignoredFilesAreSkipped() throws IOException { + Files.writeString(root.resolve(".gitignore"), "drafts\n"); + Path drafts = Files.createDirectories(root.resolve("drafts")); + Files.writeString(drafts.resolve("smith2020.yml"), ARTICLE_YAML); + Files.writeString(root.resolve("smith2020.yml"), ARTICLE_YAML); + + ScanResult result = scan(); + + assertEquals(1, result.databaseContext().getDatabase().getEntries().size()); + } + + @Test + void scanningDoesNotModifyTheDirectory() throws IOException { + Files.writeString(root.resolve("smith2020.yml"), ARTICLE_YAML); + Files.createFile(root.resolve("interesting-paper.pdf")); + List before; + try (var paths = Files.walk(root)) { + before = paths.sorted().toList(); + } + + scan(); + + List after; + try (var paths = Files.walk(root)) { + after = paths.sorted().toList(); + } + assertEquals(before, after); + } +} diff --git a/jabgui/src/test/java/org/jabref/gui/externalfiles/GitIgnoreFileFilterTest.java b/jablib/src/test/java/org/jabref/logic/util/io/GitIgnoreFileFilterTest.java similarity index 98% rename from jabgui/src/test/java/org/jabref/gui/externalfiles/GitIgnoreFileFilterTest.java rename to jablib/src/test/java/org/jabref/logic/util/io/GitIgnoreFileFilterTest.java index 467d8cdd732a..d1909e2decfd 100644 --- a/jabgui/src/test/java/org/jabref/gui/externalfiles/GitIgnoreFileFilterTest.java +++ b/jablib/src/test/java/org/jabref/logic/util/io/GitIgnoreFileFilterTest.java @@ -1,4 +1,4 @@ -package org.jabref.gui.externalfiles; +package org.jabref.logic.util.io; import java.io.IOException; import java.nio.file.Files; From 8093ffdd6f82642ef5ec25744ca5b752842c6232 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Mon, 13 Jul 2026 23:15:33 +0200 Subject: [PATCH 02/16] Link CHANGELOG entry to its pull request Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 03523016fdd9..2a1e46c9effa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv ### Added -- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear as stub entries titled after the file. Edits are not yet written back to the files. [TODO] +- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear as stub entries titled after the file. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) - The HTTP import endpoint (`POST /libraries/{id}/entries`) now accepts CSL-JSON (`application/vnd.citationstyles.csl+json`), mapping each item to the correct entry type (e.g. conference paper, book chapter, thesis) via the citation-js-based mapping. [#16151](https://github.com/JabRef/jabref/pull/16151) - We added a new "Main" tab to the entry editor showing all fields of an entry in a single scrollable list, with one-click chips for adding optional fields and a free-form box for adding arbitrary fields. Identifiers, files and links, bibliometrics, comments, and meta fields (groups, owner, timestamps, special fields) live in collapsible sections — collapsed when empty — each offering chips for its unset fields. [#12711](https://github.com/JabRef/jabref/issues/12711) - We added auto-detection import for drag-and-dropped library files. [#15391](https://github.com/JabRef/jabref/issues/15391) From f538113b923d531b43c59fc0a315a4422b1a390e Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Tue, 14 Jul 2026 02:03:20 +0200 Subject: [PATCH 03/16] Extract PDF metadata for sidecar-less PDFs in directory libraries PDFs without a sidecar now go through PdfMergeMetadataImporter (the standard PDF import pipeline: embedded BibTeX, XMP, content heuristics, plus GROBID/identifier lookups as configured) instead of becoming filename-only stubs; the stub remains the fallback when nothing usable can be extracted. The new PdfEntryFactory carries this behavior so the file watcher can reuse it. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 +- ...tory-as-library-with-hayagriva-sidecars.md | 5 +- docs/requirements/directory-library.md | 7 +- .../actions/OpenDirectoryLibraryAction.java | 5 +- .../DirectoryLibraryScanner.java | 27 ++++---- .../directorylibrary/PdfEntryFactory.java | 69 +++++++++++++++++++ .../DirectoryLibraryScannerTest.java | 33 ++++++++- 7 files changed, 127 insertions(+), 21 deletions(-) create mode 100644 jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a1e46c9effa..93968b7050ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv ### Added -- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear as stub entries titled after the file. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) +- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear as entries with metadata extracted from the PDF itself (falling back to the file name). Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) - The HTTP import endpoint (`POST /libraries/{id}/entries`) now accepts CSL-JSON (`application/vnd.citationstyles.csl+json`), mapping each item to the correct entry type (e.g. conference paper, book chapter, thesis) via the citation-js-based mapping. [#16151](https://github.com/JabRef/jabref/pull/16151) - We added a new "Main" tab to the entry editor showing all fields of an entry in a single scrollable list, with one-click chips for adding optional fields and a free-form box for adding arbitrary fields. Identifiers, files and links, bibliometrics, comments, and meta fields (groups, owner, timestamps, special fields) live in collapsible sections — collapsed when empty — each offering chips for its unset fields. [#12711](https://github.com/JabRef/jabref/issues/12711) - We added auto-detection import for drag-and-dropped library files. [#15391](https://github.com/JabRef/jabref/issues/15391) diff --git a/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md b/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md index edd9566c4b30..fb327b65e2b4 100644 --- a/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md +++ b/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md @@ -46,8 +46,9 @@ Key points of the chosen design: * Pairing is by convention — `X.yml`/`X.yaml` next to `X.pdf` — because Hayagriva has no file-path field; nothing JabRef-specific is written into the YAML for the association. * Non-Hayagriva `.yml` files (CI configs, ...) are ignored via format recognition, not reported - as errors. PDFs without a sidecar become stub entries; a sidecar is only written once the user - edits the entry (scanning never writes files). + as errors. PDFs without a sidecar become entries with metadata extracted from the PDF itself (the + standard PDF import pipeline, falling back to a stub titled after the file); a sidecar is + only written once the user edits the entry (scanning never writes files). * Later synchronization mirrors the shared-SQL seam (`convertToSharedDatabase` / `DBMSSynchronizer`): a directory synchronizer subscribes to entry events through a `CoarseChangeFilter` for write-back and applies inbound file changes with a non-local diff --git a/docs/requirements/directory-library.md b/docs/requirements/directory-library.md index 38e60f7897dd..50599ec73ac1 100644 --- a/docs/requirements/directory-library.md +++ b/docs/requirements/directory-library.md @@ -4,12 +4,13 @@ parent: Requirements # Directory library ## Directory scan builds the library from sidecars and PDFs -`req~directory-library.scan~1` +`req~directory-library.scan~2` Opening a directory as a library must fill the library from the directory tree: every Hayagriva `.yml`/`.yaml` file contributes its entries, a PDF with the same base name next to a sidecar is -linked to the sidecar's entry, and PDFs without a sidecar become stub entries titled after the -file. Hidden files/directories, gitignored paths, and `.yml` files not recognized as Hayagriva +linked to the sidecar's entry, and PDFs without a sidecar become entries with metadata +extracted from the PDF itself (embedded BibTeX, XMP, content heuristics — the standard PDF +import pipeline), falling back to a stub titled after the file. Hidden files/directories, gitignored paths, and `.yml` files not recognized as Hayagriva are skipped; unparseable Hayagriva files are reported as warnings without aborting the scan. Scanning must not write or modify any file in the directory. See [ADR 66](../decisions/0066-directory-as-library-with-hayagriva-sidecars.md) for more details. diff --git a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java index 5bb036cfbc79..de6ef6b3ec9d 100644 --- a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java +++ b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java @@ -14,6 +14,7 @@ import org.jabref.gui.util.DirectoryDialogConfiguration; import org.jabref.logic.ai.AiService; import org.jabref.logic.directorylibrary.DirectoryLibraryScanner; +import org.jabref.logic.directorylibrary.PdfEntryFactory; import org.jabref.logic.l10n.Localization; import org.jabref.logic.util.BackgroundTask; import org.jabref.logic.util.TaskExecutor; @@ -68,7 +69,9 @@ public void execute() { private void openDirectory(Path root) { preferences.getFilePreferences().setWorkingDirectory(root); - BackgroundTask.wrap(() -> new DirectoryLibraryScanner().scan(root)) + PdfEntryFactory pdfEntryFactory = new PdfEntryFactory( + preferences.getImportFormatPreferences(), preferences.getFilePreferences()); + BackgroundTask.wrap(() -> new DirectoryLibraryScanner(pdfEntryFactory).scan(root)) .onSuccess(this::showLibraryTab) .onFailure(exception -> dialogService.showErrorDialogAndWait( Localization.lang("Open folder as library"), diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java index cc9f91d99900..cb999f99b122 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java @@ -23,8 +23,6 @@ import org.jabref.model.database.BibDatabaseContext; import org.jabref.model.entry.BibEntry; import org.jabref.model.entry.LinkedFile; -import org.jabref.model.entry.field.StandardField; -import org.jabref.model.entry.types.StandardEntryType; import org.jspecify.annotations.NullMarked; import org.slf4j.Logger; @@ -32,11 +30,12 @@ /// Builds an in-memory library from a directory tree: each Hayagriva `.yml`/`.yaml` file /// contributes its entries, a PDF next to a sidecar of the same base name is linked to the -/// sidecar's (first) entry, and PDFs without a sidecar become stub entries titled after the -/// file. The directory itself is the library — the resulting [BibDatabaseContext] has -/// [org.jabref.logic.shared.DatabaseLocation#DIRECTORY] and no database path; linked files are -/// stored relative to the root, which is registered as the library-specific file directory. -// [impl->req~directory-library.scan~1] +/// sidecar's (first) entry, and PDFs without a sidecar become entries with metadata extracted +/// from the PDF itself (see [PdfEntryFactory]). The directory itself is the library — the +/// resulting [BibDatabaseContext] has [org.jabref.logic.shared.DatabaseLocation#DIRECTORY] and +/// no database path; linked files are stored relative to the root, which is registered as the +/// library-specific file directory. +// [impl->req~directory-library.scan~2] @NullMarked public class DirectoryLibraryScanner { @@ -52,6 +51,11 @@ public record ScanResult(BibDatabaseContext databaseContext, DirectoryLibraryCat private static final String PDF_EXTENSION = "pdf"; private final HayagrivaImporter importer = new HayagrivaImporter(); + private final PdfEntryFactory pdfEntryFactory; + + public DirectoryLibraryScanner(PdfEntryFactory pdfEntryFactory) { + this.pdfEntryFactory = pdfEntryFactory; + } public ScanResult scan(Path root) throws IOException { BibDatabaseContext databaseContext = new BibDatabaseContext(); @@ -92,12 +96,9 @@ public ScanResult scan(Path root) throws IOException { if (pairedPdfs.contains(pdf)) { continue; } - // No metadata extraction here: the stub only makes the PDF visible; a sidecar is - // written once the user edits the entry - BibEntry stub = new BibEntry(StandardEntryType.Misc) - .withField(StandardField.TITLE, FileUtil.getBaseName(pdf)); - linkPdf(stub, root, pdf); - entries.add(stub); + // Metadata comes from the PDF itself; a sidecar is still only written once the + // user edits the entry + entries.add(pdfEntryFactory.createEntry(pdf, root, databaseContext)); } databaseContext.getDatabase().insertEntries(entries); diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java new file mode 100644 index 000000000000..8608a49712c9 --- /dev/null +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java @@ -0,0 +1,69 @@ +package org.jabref.logic.directorylibrary; + +import java.io.IOException; +import java.nio.file.Path; +import java.util.List; +import java.util.Optional; + +import org.jabref.logic.FilePreferences; +import org.jabref.logic.importer.ImportFormatPreferences; +import org.jabref.logic.importer.ParserResult; +import org.jabref.logic.importer.fileformat.pdf.PdfMergeMetadataImporter; +import org.jabref.logic.util.StandardFileType; +import org.jabref.logic.util.io.FileUtil; +import org.jabref.model.database.BibDatabaseContext; +import org.jabref.model.entry.BibEntry; +import org.jabref.model.entry.LinkedFile; +import org.jabref.model.entry.field.StandardField; +import org.jabref.model.entry.types.StandardEntryType; + +import org.jspecify.annotations.NullMarked; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/// Creates the library entry for a PDF that has no sidecar. Metadata is extracted from the PDF +/// itself via [PdfMergeMetadataImporter] (embedded BibTeX, XMP, content heuristics, plus GROBID +/// and identifier lookups as configured — the same pipeline as dropping a PDF onto a library); +/// when nothing usable can be extracted, the entry falls back to a stub titled after the file, +/// so the PDF is at least visible. In both cases the PDF is linked relative to the library root +/// and nothing is written to disk. +@NullMarked +public class PdfEntryFactory { + + private static final Logger LOGGER = LoggerFactory.getLogger(PdfEntryFactory.class); + + private final PdfMergeMetadataImporter importer; + private final FilePreferences filePreferences; + + public PdfEntryFactory(ImportFormatPreferences importFormatPreferences, FilePreferences filePreferences) { + this.importer = new PdfMergeMetadataImporter(importFormatPreferences); + this.filePreferences = filePreferences; + } + + public BibEntry createEntry(Path pdf, Path root, BibDatabaseContext databaseContext) { + BibEntry entry = importPdfMetadata(pdf, databaseContext) + .orElseGet(() -> new BibEntry(StandardEntryType.Misc)); + if (entry.getField(StandardField.TITLE).isEmpty()) { + entry.setField(StandardField.TITLE, FileUtil.getBaseName(pdf)); + } + if (entry.getFiles().isEmpty()) { + entry.addFile(new LinkedFile("", root.relativize(pdf), StandardFileType.PDF.getName())); + } + return entry; + } + + private Optional importPdfMetadata(Path pdf, BibDatabaseContext databaseContext) { + try { + // The context overload relativizes the file link the importer attaches + ParserResult parserResult = importer.importDatabase(pdf, databaseContext, filePreferences); + List entries = parserResult.getDatabase().getEntries(); + if (parserResult.isInvalid() || entries.isEmpty()) { + return Optional.empty(); + } + return Optional.of(entries.getFirst()); + } catch (IOException e) { + LOGGER.warn("Could not extract metadata from {}", pdf, e); + return Optional.empty(); + } + } +} diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java index abc320e348c0..a42e5df9c9b1 100644 --- a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java @@ -1,12 +1,18 @@ package org.jabref.logic.directorylibrary; import java.io.IOException; +import java.net.URISyntaxException; import java.nio.file.Files; import java.nio.file.Path; import java.util.List; import java.util.Optional; +import javafx.collections.FXCollections; + +import org.jabref.logic.FilePreferences; import org.jabref.logic.directorylibrary.DirectoryLibraryScanner.ScanResult; +import org.jabref.logic.importer.ImportFormatPreferences; +import org.jabref.logic.importer.util.GrobidPreferences; import org.jabref.logic.shared.DatabaseLocation; import org.jabref.model.entry.BibEntry; import org.jabref.model.entry.LinkedFile; @@ -15,9 +21,12 @@ import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; +import org.mockito.Answers; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; class DirectoryLibraryScannerTest { @@ -33,7 +42,17 @@ class DirectoryLibraryScannerTest { @TempDir Path root; - private final DirectoryLibraryScanner scanner = new DirectoryLibraryScanner(); + private final DirectoryLibraryScanner scanner = new DirectoryLibraryScanner(offlinePdfEntryFactory()); + + /// GROBID off and no identifiers in the fixtures, so no network is touched + private static PdfEntryFactory offlinePdfEntryFactory() { + GrobidPreferences noGrobid = mock(GrobidPreferences.class, Answers.RETURNS_DEEP_STUBS); + when(noGrobid.isGrobidEnabled()).thenReturn(false); + ImportFormatPreferences importFormatPreferences = mock(ImportFormatPreferences.class, Answers.RETURNS_DEEP_STUBS); + when(importFormatPreferences.fieldPreferences().getNonWrappableFields()).thenReturn(FXCollections.emptyObservableList()); + when(importFormatPreferences.grobidPreferences()).thenReturn(noGrobid); + return new PdfEntryFactory(importFormatPreferences, mock(FilePreferences.class, Answers.RETURNS_DEEP_STUBS)); + } private ScanResult scan() throws IOException { return scanner.scan(root); @@ -106,6 +125,18 @@ void barePdfBecomesStubEntryTitledAfterTheFile() throws IOException { assertEquals(Optional.empty(), result.catalog().sourceOf(entry)); } + @Test + void barePdfMetadataIsExtractedFromTheDocument() throws IOException, URISyntaxException { + Path fixture = Path.of(getClass().getResource("/pdfs/PdfContentImporter/Kriha2018.pdf").toURI()); + Files.copy(fixture, root.resolve("kriha2018.pdf")); + + BibEntry entry = singleEntry(scan()); + + assertEquals(Optional.of("On How We Can Teach – Exploring New Ways in Professional Software Development for Students"), + entry.getField(StandardField.TITLE)); + assertEquals(List.of(new LinkedFile("", Path.of("kriha2018.pdf"), "PDF")), entry.getFiles()); + } + @Test void multiEntryFileYieldsAllEntriesRegisteredToTheSameFile() throws IOException { Files.writeString(root.resolve("collection.yml"), """ From 12fb1c37fcafa7ddea5271da7d39738e80bad327 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Tue, 14 Jul 2026 02:28:32 +0200 Subject: [PATCH 04/16] Generate citation keys for entries created from bare PDFs Entries created for PDFs without a sidecar get a citation key from the configured pattern after insertion, so the uniqueness check sees the whole library. Sidecar entries keep their Hayagriva key untouched. Co-Authored-By: Claude Fable 5 --- .../actions/OpenDirectoryLibraryAction.java | 3 +- .../DirectoryLibraryScanner.java | 6 +- .../directorylibrary/PdfEntryFactory.java | 17 ++++- .../DirectoryLibraryScannerTest.java | 21 +++++- .../directorylibrary/PdfEntryFactoryTest.java | 69 +++++++++++++++++++ 5 files changed, 112 insertions(+), 4 deletions(-) create mode 100644 jablib/src/test/java/org/jabref/logic/directorylibrary/PdfEntryFactoryTest.java diff --git a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java index de6ef6b3ec9d..88936aeba69b 100644 --- a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java +++ b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java @@ -70,7 +70,8 @@ public void execute() { private void openDirectory(Path root) { preferences.getFilePreferences().setWorkingDirectory(root); PdfEntryFactory pdfEntryFactory = new PdfEntryFactory( - preferences.getImportFormatPreferences(), preferences.getFilePreferences()); + preferences.getImportFormatPreferences(), preferences.getFilePreferences(), + preferences.getCitationKeyPatternPreferences()); BackgroundTask.wrap(() -> new DirectoryLibraryScanner(pdfEntryFactory).scan(root)) .onSuccess(this::showLibraryTab) .onFailure(exception -> dialogService.showErrorDialogAndWait( diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java index cb999f99b122..97fc2bc3d005 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java @@ -92,16 +92,20 @@ public ScanResult scan(Path root) throws IOException { entries.addAll(fileEntries); } + List pdfEntries = new ArrayList<>(); for (Path pdf : pdfFiles) { if (pairedPdfs.contains(pdf)) { continue; } // Metadata comes from the PDF itself; a sidecar is still only written once the // user edits the entry - entries.add(pdfEntryFactory.createEntry(pdf, root, databaseContext)); + pdfEntries.add(pdfEntryFactory.createEntry(pdf, root, databaseContext)); } + entries.addAll(pdfEntries); databaseContext.getDatabase().insertEntries(entries); + // After insertion, so the uniqueness check sees all scanned entries + pdfEntries.forEach(entry -> pdfEntryFactory.generateCitationKeyIfMissing(entry, databaseContext)); return new ScanResult(databaseContext, catalog, warnings); } diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java index 8608a49712c9..263e7d604814 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java @@ -6,6 +6,8 @@ import java.util.Optional; import org.jabref.logic.FilePreferences; +import org.jabref.logic.citationkeypattern.CitationKeyGenerator; +import org.jabref.logic.citationkeypattern.CitationKeyPatternPreferences; import org.jabref.logic.importer.ImportFormatPreferences; import org.jabref.logic.importer.ParserResult; import org.jabref.logic.importer.fileformat.pdf.PdfMergeMetadataImporter; @@ -34,10 +36,23 @@ public class PdfEntryFactory { private final PdfMergeMetadataImporter importer; private final FilePreferences filePreferences; + private final CitationKeyPatternPreferences citationKeyPatternPreferences; - public PdfEntryFactory(ImportFormatPreferences importFormatPreferences, FilePreferences filePreferences) { + public PdfEntryFactory(ImportFormatPreferences importFormatPreferences, + FilePreferences filePreferences, + CitationKeyPatternPreferences citationKeyPatternPreferences) { this.importer = new PdfMergeMetadataImporter(importFormatPreferences); this.filePreferences = filePreferences; + this.citationKeyPatternPreferences = citationKeyPatternPreferences; + } + + /// Generates a citation key for the entry if it has none. Call after the entry has been + /// inserted into the database, so the uniqueness check sees the whole library. + public void generateCitationKeyIfMissing(BibEntry entry, BibDatabaseContext databaseContext) { + if (entry.getCitationKey().isPresent()) { + return; + } + new CitationKeyGenerator(databaseContext, citationKeyPatternPreferences).generateAndSetKey(entry); } public BibEntry createEntry(Path pdf, Path root, BibDatabaseContext databaseContext) { diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java index a42e5df9c9b1..2287c79ebf01 100644 --- a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java @@ -7,9 +7,12 @@ import java.util.List; import java.util.Optional; +import javafx.beans.property.SimpleObjectProperty; import javafx.collections.FXCollections; import org.jabref.logic.FilePreferences; +import org.jabref.logic.citationkeypattern.CitationKeyPatternPreferences; +import org.jabref.logic.citationkeypattern.GlobalCitationKeyPatterns; import org.jabref.logic.directorylibrary.DirectoryLibraryScanner.ScanResult; import org.jabref.logic.importer.ImportFormatPreferences; import org.jabref.logic.importer.util.GrobidPreferences; @@ -23,6 +26,7 @@ import org.junit.jupiter.api.io.TempDir; import org.mockito.Answers; +import static org.jabref.logic.citationkeypattern.CitationKeyPatternPreferences.DEFAULT_UNWANTED_CHARACTERS; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.Mockito.mock; @@ -51,7 +55,22 @@ private static PdfEntryFactory offlinePdfEntryFactory() { ImportFormatPreferences importFormatPreferences = mock(ImportFormatPreferences.class, Answers.RETURNS_DEEP_STUBS); when(importFormatPreferences.fieldPreferences().getNonWrappableFields()).thenReturn(FXCollections.emptyObservableList()); when(importFormatPreferences.grobidPreferences()).thenReturn(noGrobid); - return new PdfEntryFactory(importFormatPreferences, mock(FilePreferences.class, Answers.RETURNS_DEEP_STUBS)); + return new PdfEntryFactory(importFormatPreferences, mock(FilePreferences.class, Answers.RETURNS_DEEP_STUBS), + authYearPatternPreferences()); + } + + static CitationKeyPatternPreferences authYearPatternPreferences() { + return new CitationKeyPatternPreferences( + false, + false, + false, + false, + CitationKeyPatternPreferences.KeySuffix.SECOND_WITH_A, + "", + "", + DEFAULT_UNWANTED_CHARACTERS, + GlobalCitationKeyPatterns.fromPattern("[auth][year]"), + new SimpleObjectProperty<>(',')); } private ScanResult scan() throws IOException { diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/PdfEntryFactoryTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/PdfEntryFactoryTest.java new file mode 100644 index 000000000000..5ff58f4c59d9 --- /dev/null +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/PdfEntryFactoryTest.java @@ -0,0 +1,69 @@ +package org.jabref.logic.directorylibrary; + +import java.util.List; +import java.util.Optional; + +import org.jabref.logic.FilePreferences; +import org.jabref.logic.importer.ImportFormatPreferences; +import org.jabref.model.database.BibDatabaseContext; +import org.jabref.model.entry.BibEntry; +import org.jabref.model.entry.field.StandardField; +import org.jabref.model.entry.types.StandardEntryType; + +import org.junit.jupiter.api.Test; +import org.mockito.Answers; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.mockito.Mockito.mock; + +class PdfEntryFactoryTest { + + private final PdfEntryFactory factory = new PdfEntryFactory( + mock(ImportFormatPreferences.class, Answers.RETURNS_DEEP_STUBS), + mock(FilePreferences.class, Answers.RETURNS_DEEP_STUBS), + DirectoryLibraryScannerTest.authYearPatternPreferences()); + + @Test + void generatesCitationKeyFromPattern() { + BibDatabaseContext databaseContext = new BibDatabaseContext(); + BibEntry entry = new BibEntry(StandardEntryType.Article) + .withField(StandardField.AUTHOR, "Doe, John") + .withField(StandardField.YEAR, "2016"); + databaseContext.getDatabase().insertEntry(entry); + + factory.generateCitationKeyIfMissing(entry, databaseContext); + + assertEquals(Optional.of("Doe2016"), entry.getCitationKey()); + } + + @Test + void generatedCitationKeysAreUniqueWithinTheLibrary() { + BibDatabaseContext databaseContext = new BibDatabaseContext(); + BibEntry first = new BibEntry(StandardEntryType.Article) + .withField(StandardField.AUTHOR, "Doe, John") + .withField(StandardField.YEAR, "2016") + .withCitationKey("Doe2016"); + BibEntry second = new BibEntry(StandardEntryType.Article) + .withField(StandardField.AUTHOR, "Doe, John") + .withField(StandardField.YEAR, "2016"); + databaseContext.getDatabase().insertEntries(List.of(first, second)); + + factory.generateCitationKeyIfMissing(second, databaseContext); + + assertEquals(Optional.of("Doe2016a"), second.getCitationKey()); + } + + @Test + void keepsExistingCitationKey() { + BibDatabaseContext databaseContext = new BibDatabaseContext(); + BibEntry entry = new BibEntry(StandardEntryType.Article) + .withField(StandardField.AUTHOR, "Doe, John") + .withField(StandardField.YEAR, "2016") + .withCitationKey("customKey"); + databaseContext.getDatabase().insertEntry(entry); + + factory.generateCitationKeyIfMissing(entry, databaseContext); + + assertEquals(Optional.of("customKey"), entry.getCitationKey()); + } +} From bca0a6fab1960707605706aaf449ef507f1438f1 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Tue, 14 Jul 2026 02:51:19 +0200 Subject: [PATCH 05/16] Extract PDF metadata asynchronously after the directory library opens Opening a folder no longer blocks on PDF parsing or identifier lookups: the scan inserts quick stub entries for sidecar-less PDFs and the new PdfEnrichmentTask extracts their metadata in the background afterwards, enriching each stub in place (progress shown in the task overlay, cancellable). Enrichment mutations carry EntriesEventSource.SHARED because they are system-initiated, not user edits; the citation key is generated per entry once its metadata arrived. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 +- ...tory-as-library-with-hayagriva-sidecars.md | 7 +- docs/requirements/directory-library.md | 9 +-- .../actions/OpenDirectoryLibraryAction.java | 11 ++++ .../DirectoryLibraryScanner.java | 29 +++++---- .../directorylibrary/PdfEnrichmentTask.java | 64 +++++++++++++++++++ .../directorylibrary/PdfEntryFactory.java | 27 +++++++- .../main/resources/l10n/JabRef_en.properties | 1 + .../DirectoryLibraryScannerTest.java | 12 +++- 9 files changed, 138 insertions(+), 24 deletions(-) create mode 100644 jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java diff --git a/CHANGELOG.md b/CHANGELOG.md index 93968b7050ca..3e399e9e7447 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv ### Added -- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear as entries with metadata extracted from the PDF itself (falling back to the file name). Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) +- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear immediately and are enriched in the background with metadata extracted from the PDF itself and a generated citation key. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) - The HTTP import endpoint (`POST /libraries/{id}/entries`) now accepts CSL-JSON (`application/vnd.citationstyles.csl+json`), mapping each item to the correct entry type (e.g. conference paper, book chapter, thesis) via the citation-js-based mapping. [#16151](https://github.com/JabRef/jabref/pull/16151) - We added a new "Main" tab to the entry editor showing all fields of an entry in a single scrollable list, with one-click chips for adding optional fields and a free-form box for adding arbitrary fields. Identifiers, files and links, bibliometrics, comments, and meta fields (groups, owner, timestamps, special fields) live in collapsible sections — collapsed when empty — each offering chips for its unset fields. [#12711](https://github.com/JabRef/jabref/issues/12711) - We added auto-detection import for drag-and-dropped library files. [#15391](https://github.com/JabRef/jabref/issues/15391) diff --git a/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md b/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md index fb327b65e2b4..1092e58d9248 100644 --- a/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md +++ b/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md @@ -46,9 +46,10 @@ Key points of the chosen design: * Pairing is by convention — `X.yml`/`X.yaml` next to `X.pdf` — because Hayagriva has no file-path field; nothing JabRef-specific is written into the YAML for the association. * Non-Hayagriva `.yml` files (CI configs, ...) are ignored via format recognition, not reported - as errors. PDFs without a sidecar become entries with metadata extracted from the PDF itself (the - standard PDF import pipeline, falling back to a stub titled after the file); a sidecar is - only written once the user edits the entry (scanning never writes files). + as errors. PDFs without a sidecar appear immediately as stubs; their metadata is extracted + asynchronously after the library is shown (the standard PDF import pipeline, enriching the + stub in place); a sidecar is only written once the user edits the entry (scanning never + writes files). * Later synchronization mirrors the shared-SQL seam (`convertToSharedDatabase` / `DBMSSynchronizer`): a directory synchronizer subscribes to entry events through a `CoarseChangeFilter` for write-back and applies inbound file changes with a non-local diff --git a/docs/requirements/directory-library.md b/docs/requirements/directory-library.md index 50599ec73ac1..6064e254b0f9 100644 --- a/docs/requirements/directory-library.md +++ b/docs/requirements/directory-library.md @@ -4,13 +4,14 @@ parent: Requirements # Directory library ## Directory scan builds the library from sidecars and PDFs -`req~directory-library.scan~2` +`req~directory-library.scan~3` Opening a directory as a library must fill the library from the directory tree: every Hayagriva `.yml`/`.yaml` file contributes its entries, a PDF with the same base name next to a sidecar is -linked to the sidecar's entry, and PDFs without a sidecar become entries with metadata -extracted from the PDF itself (embedded BibTeX, XMP, content heuristics — the standard PDF -import pipeline), falling back to a stub titled after the file. Hidden files/directories, gitignored paths, and `.yml` files not recognized as Hayagriva +linked to the sidecar's entry, and PDFs without a sidecar appear immediately as stub entries +titled after the file; their metadata (embedded BibTeX, XMP, content heuristics — the standard +PDF import pipeline) and a generated citation key arrive asynchronously after the library is +shown, without replacing the entry instances. Opening must not block on PDF parsing. Hidden files/directories, gitignored paths, and `.yml` files not recognized as Hayagriva are skipped; unparseable Hayagriva files are reported as warnings without aborting the scan. Scanning must not write or modify any file in the directory. See [ADR 66](../decisions/0066-directory-as-library-with-hayagriva-sidecars.md) for more details. diff --git a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java index 88936aeba69b..3223e551dc0d 100644 --- a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java +++ b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java @@ -12,8 +12,10 @@ import org.jabref.gui.clipboard.ClipBoardManager; import org.jabref.gui.preferences.GuiPreferences; import org.jabref.gui.util.DirectoryDialogConfiguration; +import org.jabref.gui.util.UiTaskExecutor; import org.jabref.logic.ai.AiService; import org.jabref.logic.directorylibrary.DirectoryLibraryScanner; +import org.jabref.logic.directorylibrary.PdfEnrichmentTask; import org.jabref.logic.directorylibrary.PdfEntryFactory; import org.jabref.logic.l10n.Localization; import org.jabref.logic.util.BackgroundTask; @@ -97,6 +99,15 @@ private void showLibraryTab(DirectoryLibraryScanner.ScanResult scanResult) { clipBoardManager, taskExecutor); tabContainer.addTab(libraryTab, true); + if (!scanResult.pendingPdfImports().isEmpty()) { + PdfEntryFactory pdfEntryFactory = new PdfEntryFactory( + preferences.getImportFormatPreferences(), preferences.getFilePreferences(), + preferences.getCitationKeyPatternPreferences()); + new PdfEnrichmentTask(scanResult.pendingPdfImports(), pdfEntryFactory, scanResult.databaseContext(), + UiTaskExecutor::runInJavaFXThread) + .executeWith(taskExecutor); + } + if (!scanResult.warnings().isEmpty()) { dialogService.showWarningDialogAndWait( Localization.lang("Open folder as library"), diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java index 97fc2bc3d005..e04cef66f8d4 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java @@ -35,14 +35,20 @@ /// resulting [BibDatabaseContext] has [org.jabref.logic.shared.DatabaseLocation#DIRECTORY] and /// no database path; linked files are stored relative to the root, which is registered as the /// library-specific file directory. -// [impl->req~directory-library.scan~2] +// [impl->req~directory-library.scan~3] @NullMarked public class DirectoryLibraryScanner { /// Everything a directory scan produces: the ready-to-open context, the entry-to-file - /// catalog (consumed by the file synchronization in later steps), and user-facing warnings - /// about files that looked like Hayagriva but could not be parsed. - public record ScanResult(BibDatabaseContext databaseContext, DirectoryLibraryCatalog catalog, List warnings) { + /// catalog (consumed by the file synchronization in later steps), user-facing warnings + /// about files that looked like Hayagriva but could not be parsed, and the sidecar-less + /// PDFs whose stub entries still await metadata extraction (see [PdfEnrichmentTask]). + public record ScanResult(BibDatabaseContext databaseContext, DirectoryLibraryCatalog catalog, List warnings, + List pendingPdfImports) { + } + + /// A stub entry created for a sidecar-less PDF, awaiting asynchronous metadata extraction. + public record PendingPdfImport(BibEntry entry, Path pdfFile) { } private static final Logger LOGGER = LoggerFactory.getLogger(DirectoryLibraryScanner.class); @@ -92,21 +98,20 @@ public ScanResult scan(Path root) throws IOException { entries.addAll(fileEntries); } - List pdfEntries = new ArrayList<>(); + List pendingPdfImports = new ArrayList<>(); for (Path pdf : pdfFiles) { if (pairedPdfs.contains(pdf)) { continue; } - // Metadata comes from the PDF itself; a sidecar is still only written once the - // user edits the entry - pdfEntries.add(pdfEntryFactory.createEntry(pdf, root, databaseContext)); + // Only a quick stub here: metadata extraction happens asynchronously after the + // library is shown, and a sidecar is still only written once the user edits + BibEntry stub = pdfEntryFactory.createStub(pdf, root); + pendingPdfImports.add(new PendingPdfImport(stub, pdf)); + entries.add(stub); } - entries.addAll(pdfEntries); databaseContext.getDatabase().insertEntries(entries); - // After insertion, so the uniqueness check sees all scanned entries - pdfEntries.forEach(entry -> pdfEntryFactory.generateCitationKeyIfMissing(entry, databaseContext)); - return new ScanResult(databaseContext, catalog, warnings); + return new ScanResult(databaseContext, catalog, warnings, List.copyOf(pendingPdfImports)); } private void collectFiles(Path root, List yamlFiles, List pdfFiles) throws IOException { diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java new file mode 100644 index 000000000000..9db8c0482763 --- /dev/null +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java @@ -0,0 +1,64 @@ +package org.jabref.logic.directorylibrary; + +import java.util.List; +import java.util.Optional; +import java.util.function.Consumer; + +import org.jabref.logic.l10n.Localization; +import org.jabref.logic.util.BackgroundTask; +import org.jabref.model.database.BibDatabaseContext; +import org.jabref.model.entry.BibEntry; + +import org.jspecify.annotations.NullMarked; + +/// Extracts metadata for the sidecar-less PDFs of a directory library after the tab is already +/// visible: each pending stub entry is enriched in place as its PDF is processed, so the table +/// fills progressively and opening the folder never blocks on PDF parsing or identifier +/// lookups. Cancelling keeps the stubs created by the scan. +/// +/// Extraction runs on the background thread; entry mutations (and progress updates, which drive +/// JavaFX properties) go through the injected marshaller. +// [impl->req~directory-library.scan~3] +@NullMarked +public class PdfEnrichmentTask extends BackgroundTask { + + private final List pendingImports; + private final PdfEntryFactory pdfEntryFactory; + private final BibDatabaseContext databaseContext; + private final Consumer modelUpdateMarshaller; + + public PdfEnrichmentTask(List pendingImports, + PdfEntryFactory pdfEntryFactory, + BibDatabaseContext databaseContext, + Consumer modelUpdateMarshaller) { + this.pendingImports = pendingImports; + this.pdfEntryFactory = pdfEntryFactory; + this.databaseContext = databaseContext; + this.modelUpdateMarshaller = modelUpdateMarshaller; + setTitle(Localization.lang("Extracting metadata from %0 PDF file(s)", Integer.toString(pendingImports.size()))); + showToUser(true); + } + + @Override + public Void call() { + int counter = 0; + for (DirectoryLibraryScanner.PendingPdfImport pending : pendingImports) { + if (isCancelled()) { + break; + } + counter++; + final int progress = counter; + modelUpdateMarshaller.accept(() -> { + updateMessage(pending.pdfFile().getFileName().toString()); + updateProgress(progress, pendingImports.size()); + }); + + Optional extracted = pdfEntryFactory.extractMetadata(pending.pdfFile(), databaseContext); + modelUpdateMarshaller.accept(() -> { + extracted.ifPresent(metadata -> pdfEntryFactory.applyExtractedMetadata(metadata, pending.entry())); + pdfEntryFactory.generateCitationKeyIfMissing(pending.entry(), databaseContext); + }); + } + return null; + } +} diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java index 263e7d604814..9c56771a541a 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java @@ -16,6 +16,7 @@ import org.jabref.model.database.BibDatabaseContext; import org.jabref.model.entry.BibEntry; import org.jabref.model.entry.LinkedFile; +import org.jabref.model.entry.event.EntriesEventSource; import org.jabref.model.entry.field.StandardField; import org.jabref.model.entry.types.StandardEntryType; @@ -56,7 +57,7 @@ public void generateCitationKeyIfMissing(BibEntry entry, BibDatabaseContext data } public BibEntry createEntry(Path pdf, Path root, BibDatabaseContext databaseContext) { - BibEntry entry = importPdfMetadata(pdf, databaseContext) + BibEntry entry = extractMetadata(pdf, databaseContext) .orElseGet(() -> new BibEntry(StandardEntryType.Misc)); if (entry.getField(StandardField.TITLE).isEmpty()) { entry.setField(StandardField.TITLE, FileUtil.getBaseName(pdf)); @@ -67,7 +68,29 @@ public BibEntry createEntry(Path pdf, Path root, BibDatabaseContext databaseCont return entry; } - private Optional importPdfMetadata(Path pdf, BibDatabaseContext databaseContext) { + /// The immediately available placeholder for a PDF: title from the file name, PDF linked. + /// Metadata extraction happens asynchronously afterwards (see [PdfEnrichmentTask]). + public BibEntry createStub(Path pdf, Path root) { + BibEntry stub = new BibEntry(StandardEntryType.Misc) + .withField(StandardField.TITLE, FileUtil.getBaseName(pdf)); + stub.addFile(new LinkedFile("", root.relativize(pdf), StandardFileType.PDF.getName())); + return stub; + } + + /// Applies extracted metadata onto the live stub entry (same instance, so table selection + /// survives). Mutations carry [EntriesEventSource#SHARED]: enrichment is system-initiated, + /// not a user edit. The stub's file link is preserved. + public void applyExtractedMetadata(BibEntry extracted, BibEntry stub) { + if (!stub.getType().equals(extracted.getType())) { + stub.setType(extracted.getType(), EntriesEventSource.SHARED); + } + extracted.getFields().stream() + .filter(field -> StandardField.FILE != field) + .forEach(field -> extracted.getField(field).ifPresent( + value -> stub.setField(field, value, EntriesEventSource.SHARED))); + } + + public Optional extractMetadata(Path pdf, BibDatabaseContext databaseContext) { try { // The context overload relativizes the file link the importer attaches ParserResult parserResult = importer.importDatabase(pdf, databaseContext, filePreferences); diff --git a/jablib/src/main/resources/l10n/JabRef_en.properties b/jablib/src/main/resources/l10n/JabRef_en.properties index 9abfc8c63c66..7b2911fdabf2 100644 --- a/jablib/src/main/resources/l10n/JabRef_en.properties +++ b/jablib/src/main/resources/l10n/JabRef_en.properties @@ -763,6 +763,7 @@ Select\ entry\ to\ link=Select entry to link new=new Open\ file=Open file +Extracting\ metadata\ from\ %0\ PDF\ file(s)=Extracting metadata from %0 PDF file(s) Open\ folder\ as\ library=Open folder as library Open\ folder\ as\ library...=Open folder as library... Error\ opening\ file=Error opening file diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java index 2287c79ebf01..603ebf4d8778 100644 --- a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java @@ -142,14 +142,22 @@ void barePdfBecomesStubEntryTitledAfterTheFile() throws IOException { assertEquals(Optional.of("interesting-paper"), entry.getField(StandardField.TITLE)); assertEquals(List.of(new LinkedFile("", Path.of("interesting-paper.pdf"), "PDF")), entry.getFiles()); assertEquals(Optional.empty(), result.catalog().sourceOf(entry)); + assertEquals(List.of(new DirectoryLibraryScanner.PendingPdfImport(entry, root.resolve("interesting-paper.pdf"))), + result.pendingPdfImports()); } @Test - void barePdfMetadataIsExtractedFromTheDocument() throws IOException, URISyntaxException { + void enrichmentExtractsPdfMetadataIntoTheStubEntry() throws IOException, URISyntaxException { Path fixture = Path.of(getClass().getResource("/pdfs/PdfContentImporter/Kriha2018.pdf").toURI()); Files.copy(fixture, root.resolve("kriha2018.pdf")); - BibEntry entry = singleEntry(scan()); + ScanResult result = scan(); + BibEntry entry = singleEntry(result); + // Scanning is instant: only the stub exists until the enrichment task ran + assertEquals(Optional.of("kriha2018"), entry.getField(StandardField.TITLE)); + + new PdfEnrichmentTask(result.pendingPdfImports(), offlinePdfEntryFactory(), + result.databaseContext(), Runnable::run).call(); assertEquals(Optional.of("On How We Can Teach – Exploring New Ways in Professional Software Development for Students"), entry.getField(StandardField.TITLE)); From d23a044402857f474c7e4332f5c55ee22054b734 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Tue, 14 Jul 2026 04:10:58 +0200 Subject: [PATCH 06/16] Look up missing DOIs during PDF metadata extraction When the PDF itself yields no DOI, the DOI is looked up via CrossRef (the same reverse search as the magnifier button next to the DOI field) and the metadata behind it merges into the entry with set-if-empty semantics (BibEntry#mergeWith): PDF-extracted values are never overwritten, the DOI metadata only fills the gaps. PDFs that already carry a DOI keep being enriched by PdfMergeMetadataImporter itself. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 +- docs/requirements/directory-library.md | 5 +- .../DirectoryLibraryScanner.java | 2 +- .../directorylibrary/PdfEnrichmentTask.java | 2 +- .../directorylibrary/PdfEntryFactory.java | 41 ++++++++++++++++- .../DirectoryLibraryScannerTest.java | 4 +- .../directorylibrary/PdfEntryFactoryTest.java | 46 ++++++++++++++++++- 7 files changed, 93 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3e399e9e7447..baa6e992806d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv ### Added -- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear immediately and are enriched in the background with metadata extracted from the PDF itself and a generated citation key. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) +- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear immediately and are enriched in the background with metadata extracted from the PDF itself and a generated citation key; a missing DOI is looked up online and its metadata fills the remaining empty fields. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) - The HTTP import endpoint (`POST /libraries/{id}/entries`) now accepts CSL-JSON (`application/vnd.citationstyles.csl+json`), mapping each item to the correct entry type (e.g. conference paper, book chapter, thesis) via the citation-js-based mapping. [#16151](https://github.com/JabRef/jabref/pull/16151) - We added a new "Main" tab to the entry editor showing all fields of an entry in a single scrollable list, with one-click chips for adding optional fields and a free-form box for adding arbitrary fields. Identifiers, files and links, bibliometrics, comments, and meta fields (groups, owner, timestamps, special fields) live in collapsible sections — collapsed when empty — each offering chips for its unset fields. [#12711](https://github.com/JabRef/jabref/issues/12711) - We added auto-detection import for drag-and-dropped library files. [#15391](https://github.com/JabRef/jabref/issues/15391) diff --git a/docs/requirements/directory-library.md b/docs/requirements/directory-library.md index 6064e254b0f9..d3f0d033f135 100644 --- a/docs/requirements/directory-library.md +++ b/docs/requirements/directory-library.md @@ -4,14 +4,15 @@ parent: Requirements # Directory library ## Directory scan builds the library from sidecars and PDFs -`req~directory-library.scan~3` +`req~directory-library.scan~4` Opening a directory as a library must fill the library from the directory tree: every Hayagriva `.yml`/`.yaml` file contributes its entries, a PDF with the same base name next to a sidecar is linked to the sidecar's entry, and PDFs without a sidecar appear immediately as stub entries titled after the file; their metadata (embedded BibTeX, XMP, content heuristics — the standard PDF import pipeline) and a generated citation key arrive asynchronously after the library is -shown, without replacing the entry instances. Opening must not block on PDF parsing. Hidden files/directories, gitignored paths, and `.yml` files not recognized as Hayagriva +shown, without replacing the entry instances. When the PDF yields no DOI, the DOI is looked up +online and the metadata behind it fills only the fields the PDF did not provide. Opening must not block on PDF parsing. Hidden files/directories, gitignored paths, and `.yml` files not recognized as Hayagriva are skipped; unparseable Hayagriva files are reported as warnings without aborting the scan. Scanning must not write or modify any file in the directory. See [ADR 66](../decisions/0066-directory-as-library-with-hayagriva-sidecars.md) for more details. diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java index e04cef66f8d4..1bb5c9472841 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java @@ -35,7 +35,7 @@ /// resulting [BibDatabaseContext] has [org.jabref.logic.shared.DatabaseLocation#DIRECTORY] and /// no database path; linked files are stored relative to the root, which is registered as the /// library-specific file directory. -// [impl->req~directory-library.scan~3] +// [impl->req~directory-library.scan~4] @NullMarked public class DirectoryLibraryScanner { diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java index 9db8c0482763..fc6755d3072e 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java @@ -18,7 +18,7 @@ /// /// Extraction runs on the background thread; entry mutations (and progress updates, which drive /// JavaFX properties) go through the injected marshaller. -// [impl->req~directory-library.scan~3] +// [impl->req~directory-library.scan~4] @NullMarked public class PdfEnrichmentTask extends BackgroundTask { diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java index 9c56771a541a..391456b39d1a 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java @@ -8,8 +8,11 @@ import org.jabref.logic.FilePreferences; import org.jabref.logic.citationkeypattern.CitationKeyGenerator; import org.jabref.logic.citationkeypattern.CitationKeyPatternPreferences; +import org.jabref.logic.importer.FetcherException; import org.jabref.logic.importer.ImportFormatPreferences; import org.jabref.logic.importer.ParserResult; +import org.jabref.logic.importer.fetcher.CrossRef; +import org.jabref.logic.importer.fetcher.DoiFetcher; import org.jabref.logic.importer.fileformat.pdf.PdfMergeMetadataImporter; import org.jabref.logic.util.StandardFileType; import org.jabref.logic.util.io.FileUtil; @@ -18,6 +21,7 @@ import org.jabref.model.entry.LinkedFile; import org.jabref.model.entry.event.EntriesEventSource; import org.jabref.model.entry.field.StandardField; +import org.jabref.model.entry.identifier.DOI; import org.jabref.model.entry.types.StandardEntryType; import org.jspecify.annotations.NullMarked; @@ -38,13 +42,26 @@ public class PdfEntryFactory { private final PdfMergeMetadataImporter importer; private final FilePreferences filePreferences; private final CitationKeyPatternPreferences citationKeyPatternPreferences; + private final CrossRef crossRef; + private final DoiFetcher doiFetcher; public PdfEntryFactory(ImportFormatPreferences importFormatPreferences, FilePreferences filePreferences, CitationKeyPatternPreferences citationKeyPatternPreferences) { + this(importFormatPreferences, filePreferences, citationKeyPatternPreferences, + new CrossRef(), new DoiFetcher(importFormatPreferences)); + } + + PdfEntryFactory(ImportFormatPreferences importFormatPreferences, + FilePreferences filePreferences, + CitationKeyPatternPreferences citationKeyPatternPreferences, + CrossRef crossRef, + DoiFetcher doiFetcher) { this.importer = new PdfMergeMetadataImporter(importFormatPreferences); this.filePreferences = filePreferences; this.citationKeyPatternPreferences = citationKeyPatternPreferences; + this.crossRef = crossRef; + this.doiFetcher = doiFetcher; } /// Generates a citation key for the entry if it has none. Call after the entry has been @@ -98,10 +115,32 @@ public Optional extractMetadata(Path pdf, BibDatabaseContext databaseC if (parserResult.isInvalid() || entries.isEmpty()) { return Optional.empty(); } - return Optional.of(entries.getFirst()); + BibEntry entry = entries.getFirst(); + completeWithMissingDoiMetadata(entry); + return Optional.of(entry); } catch (IOException e) { LOGGER.warn("Could not extract metadata from {}", pdf, e); return Optional.empty(); } } + + /// When the PDF itself yielded no DOI, the DOI is looked up (the same reverse search as the + /// magnifier button next to the DOI field) and the metadata behind it fills the fields the + /// PDF did not provide — extracted values are never overwritten. PDFs that already carried a + /// DOI were enriched by [PdfMergeMetadataImporter] itself. + private void completeWithMissingDoiMetadata(BibEntry entry) { + if (entry.getField(StandardField.DOI).isPresent()) { + return; + } + try { + Optional doi = crossRef.findIdentifier(entry); + if (doi.isEmpty()) { + return; + } + entry.setField(StandardField.DOI, doi.get().asString()); + doiFetcher.performSearchById(doi.get().asString()).ifPresent(entry::mergeWith); + } catch (FetcherException e) { + LOGGER.debug("DOI lookup failed for {}", entry.getAuthorTitleYear(80), e); + } + } } diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java index 603ebf4d8778..0cf3bef69d02 100644 --- a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java @@ -15,6 +15,8 @@ import org.jabref.logic.citationkeypattern.GlobalCitationKeyPatterns; import org.jabref.logic.directorylibrary.DirectoryLibraryScanner.ScanResult; import org.jabref.logic.importer.ImportFormatPreferences; +import org.jabref.logic.importer.fetcher.CrossRef; +import org.jabref.logic.importer.fetcher.DoiFetcher; import org.jabref.logic.importer.util.GrobidPreferences; import org.jabref.logic.shared.DatabaseLocation; import org.jabref.model.entry.BibEntry; @@ -56,7 +58,7 @@ private static PdfEntryFactory offlinePdfEntryFactory() { when(importFormatPreferences.fieldPreferences().getNonWrappableFields()).thenReturn(FXCollections.emptyObservableList()); when(importFormatPreferences.grobidPreferences()).thenReturn(noGrobid); return new PdfEntryFactory(importFormatPreferences, mock(FilePreferences.class, Answers.RETURNS_DEEP_STUBS), - authYearPatternPreferences()); + authYearPatternPreferences(), mock(CrossRef.class), mock(DoiFetcher.class)); } static CitationKeyPatternPreferences authYearPatternPreferences() { diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/PdfEntryFactoryTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/PdfEntryFactoryTest.java index 5ff58f4c59d9..17d151d49523 100644 --- a/jablib/src/test/java/org/jabref/logic/directorylibrary/PdfEntryFactoryTest.java +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/PdfEntryFactoryTest.java @@ -1,27 +1,69 @@ package org.jabref.logic.directorylibrary; +import java.net.URISyntaxException; +import java.nio.file.Path; import java.util.List; import java.util.Optional; +import javafx.collections.FXCollections; + import org.jabref.logic.FilePreferences; +import org.jabref.logic.importer.FetcherException; import org.jabref.logic.importer.ImportFormatPreferences; +import org.jabref.logic.importer.fetcher.CrossRef; +import org.jabref.logic.importer.fetcher.DoiFetcher; +import org.jabref.logic.importer.util.GrobidPreferences; import org.jabref.model.database.BibDatabaseContext; import org.jabref.model.entry.BibEntry; import org.jabref.model.entry.field.StandardField; +import org.jabref.model.entry.identifier.DOI; import org.jabref.model.entry.types.StandardEntryType; import org.junit.jupiter.api.Test; import org.mockito.Answers; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.mockito.Mockito.any; import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; class PdfEntryFactoryTest { + private final CrossRef crossRef = mock(CrossRef.class); + private final DoiFetcher doiFetcher = mock(DoiFetcher.class); private final PdfEntryFactory factory = new PdfEntryFactory( - mock(ImportFormatPreferences.class, Answers.RETURNS_DEEP_STUBS), + offlineImportFormatPreferences(), mock(FilePreferences.class, Answers.RETURNS_DEEP_STUBS), - DirectoryLibraryScannerTest.authYearPatternPreferences()); + DirectoryLibraryScannerTest.authYearPatternPreferences(), + crossRef, + doiFetcher); + + private static ImportFormatPreferences offlineImportFormatPreferences() { + GrobidPreferences noGrobid = mock(GrobidPreferences.class, Answers.RETURNS_DEEP_STUBS); + when(noGrobid.isGrobidEnabled()).thenReturn(false); + ImportFormatPreferences importFormatPreferences = mock(ImportFormatPreferences.class, Answers.RETURNS_DEEP_STUBS); + when(importFormatPreferences.fieldPreferences().getNonWrappableFields()).thenReturn(FXCollections.emptyObservableList()); + when(importFormatPreferences.grobidPreferences()).thenReturn(noGrobid); + return importFormatPreferences; + } + + @Test + void missingDoiIsLookedUpAndItsMetadataFillsOnlyEmptyFields() throws FetcherException, URISyntaxException { + when(crossRef.findIdentifier(any())).thenReturn(Optional.of(DOI.parse("10.1000/demo").orElseThrow())); + BibEntry fetched = new BibEntry(StandardEntryType.Article) + .withField(StandardField.TITLE, "Fetched Title Must Not Win") + .withField(StandardField.JOURNAL, "Fetched Journal"); + when(doiFetcher.performSearchById("10.1000/demo")).thenReturn(Optional.of(fetched)); + Path pdf = Path.of(getClass().getResource("/pdfs/PdfContentImporter/Kriha2018.pdf").toURI()); + + BibEntry extracted = factory.extractMetadata(pdf, new BibDatabaseContext()).orElseThrow(); + + // The PDF-extracted title has priority; DOI metadata only fills gaps + assertEquals(Optional.of("On How We Can Teach – Exploring New Ways in Professional Software Development for Students"), + extracted.getField(StandardField.TITLE)); + assertEquals(Optional.of("Fetched Journal"), extracted.getField(StandardField.JOURNAL)); + assertEquals(Optional.of("10.1000/demo"), extracted.getField(StandardField.DOI)); + } @Test void generatesCitationKeyFromPattern() { From b3cb4dc195acf828d749e85ad1c70bf2ca316cd6 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Tue, 14 Jul 2026 04:20:30 +0200 Subject: [PATCH 07/16] Restore directory libraries with the last session Directory libraries were skipped by the session store because they have no database path; their root directory now stands in, and paths that turn out to be directories are routed from openFiles to the directory-library opener on the next start. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 +- docs/requirements/directory-library.md | 9 +++++++++ .../org/jabref/gui/frame/JabRefFrameViewModel.java | 8 +++++--- .../gui/importer/actions/OpenDatabaseAction.java | 10 ++++++++++ .../importer/actions/OpenDirectoryLibraryAction.java | 4 +++- 5 files changed, 28 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index baa6e992806d..fd5680c88155 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv ### Added -- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear immediately and are enriched in the background with metadata extracted from the PDF itself and a generated citation key; a missing DOI is looked up online and its metadata fills the remaining empty fields. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) +- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear immediately and are enriched in the background with metadata extracted from the PDF itself and a generated citation key; a missing DOI is looked up online and its metadata fills the remaining empty fields. Directory libraries that were open on shutdown are reopened on the next start. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) - The HTTP import endpoint (`POST /libraries/{id}/entries`) now accepts CSL-JSON (`application/vnd.citationstyles.csl+json`), mapping each item to the correct entry type (e.g. conference paper, book chapter, thesis) via the citation-js-based mapping. [#16151](https://github.com/JabRef/jabref/pull/16151) - We added a new "Main" tab to the entry editor showing all fields of an entry in a single scrollable list, with one-click chips for adding optional fields and a free-form box for adding arbitrary fields. Identifiers, files and links, bibliometrics, comments, and meta fields (groups, owner, timestamps, special fields) live in collapsible sections — collapsed when empty — each offering chips for its unset fields. [#12711](https://github.com/JabRef/jabref/issues/12711) - We added auto-detection import for drag-and-dropped library files. [#15391](https://github.com/JabRef/jabref/issues/15391) diff --git a/docs/requirements/directory-library.md b/docs/requirements/directory-library.md index d3f0d033f135..ffad2d8f861d 100644 --- a/docs/requirements/directory-library.md +++ b/docs/requirements/directory-library.md @@ -19,4 +19,13 @@ See [ADR 66](../decisions/0066-directory-as-library-with-hayagriva-sidecars.md) Needs: impl +## Directory libraries are part of the restored session +`req~directory-library.session-restore~1` + +When "Open last edited libraries" is enabled, a directory library that was open on shutdown is +reopened on the next start, exactly like `.bib` libraries: its root directory is remembered in +the last-opened list and routed back through the directory-library opener. + +Needs: impl + diff --git a/jabgui/src/main/java/org/jabref/gui/frame/JabRefFrameViewModel.java b/jabgui/src/main/java/org/jabref/gui/frame/JabRefFrameViewModel.java index f9aac3c1b446..39669e6ee4ac 100644 --- a/jabgui/src/main/java/org/jabref/gui/frame/JabRefFrameViewModel.java +++ b/jabgui/src/main/java/org/jabref/gui/frame/JabRefFrameViewModel.java @@ -129,16 +129,18 @@ public boolean close() { } } - // Read the opened and focused databases before closing them + // Read the opened and focused databases before closing them. Directory libraries have + // no database path; their root stands in so they are restored on the next start. + // [impl->req~directory-library.session-restore~1] List openedLibraries = tabContainer.getLibraryTabs().stream() .map(LibraryTab::getBibDatabaseContext) - .map(BibDatabaseContext::getDatabasePath) + .map(context -> context.getDatabasePath().or(context::getDirectoryLibraryRoot)) .flatMap(Optional::stream) .map(Path::toAbsolutePath) .toList(); Path focusedLibraries = Optional.ofNullable(tabContainer.getCurrentLibraryTab()) .map(LibraryTab::getBibDatabaseContext) - .flatMap(BibDatabaseContext::getDatabasePath) + .flatMap(context -> context.getDatabasePath().or(context::getDirectoryLibraryRoot)) .map(Path::toAbsolutePath) .orElse(null); diff --git a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDatabaseAction.java b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDatabaseAction.java index 106bde5be6a5..7145005987fc 100644 --- a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDatabaseAction.java +++ b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDatabaseAction.java @@ -165,6 +165,16 @@ public void openFile(Path file) { /// /// @param filesToOpen the filesToOpen, may be null or not existing public void openFiles(List filesToOpen) { + // Directories are directory libraries (e.g. restored from the last session) + // [impl->req~directory-library.session-restore~1] + filesToOpen.stream() + .map(FileUtil::resolveIfShortcut) + .filter(Files::isDirectory) + .distinct() + .forEach(directory -> new OpenDirectoryLibraryAction(tabContainer, dialogService, preferences, + aiService, stateManager, fileUpdateMonitor, entryTypesManager, undoManager, + clipboardManager, taskExecutor).openDirectory(directory)); + // Resolve any shortcuts to their targets and filter to only .bib files. // The resulting list must remain modifiable for downstream processing (iterator.remove() calls below). Path baseDirectoryPath = JabRefBaseDirectoryLocator.getBaseDirectoryPath(); diff --git a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java index 3223e551dc0d..5f6ea72965e0 100644 --- a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java +++ b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java @@ -69,7 +69,9 @@ public void execute() { .ifPresent(this::openDirectory); } - private void openDirectory(Path root) { + /// Opens the directory as a library without showing the chooser (used by the session + /// restore and internal routing). + public void openDirectory(Path root) { preferences.getFilePreferences().setWorkingDirectory(root); PdfEntryFactory pdfEntryFactory = new PdfEntryFactory( preferences.getImportFormatPreferences(), preferences.getFilePreferences(), From 9f434e7afc77ddb8bd6dbf327293ce04e91fdbdc Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Fri, 17 Jul 2026 01:45:24 +0200 Subject: [PATCH 08/16] Read Markdown sidecars with Hayagriva frontmatter The directory-library spec gains a Markdown sidecar form: X.md next to X.pdf, whose YAML frontmatter is a regular Hayagriva document and whose body carries the long-form notes (# Notes for the comment field, one ## comment- section per per-user comment). Long-form Markdown notes read terribly as YAML block scalars; as a Markdown body the folder doubles as a plain notes collection (Obsidian, any editor). The scanner imports such files next to plain .yml/.yaml Hayagriva files. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Q5Y5GTihE9rZxKrvPqnHV4 --- CHANGELOG.md | 2 +- ...tory-as-library-with-hayagriva-sidecars.md | 39 +++-- docs/requirements/directory-library.md | 10 +- .../DirectoryLibraryScanner.java | 50 +++--- .../directorylibrary/MarkdownSidecar.java | 134 +++++++++++++++ .../directorylibrary/PdfEnrichmentTask.java | 2 +- .../DirectoryLibraryScannerTest.java | 47 ++++++ .../directorylibrary/MarkdownSidecarTest.java | 152 ++++++++++++++++++ 8 files changed, 398 insertions(+), 38 deletions(-) create mode 100644 jablib/src/main/java/org/jabref/logic/directorylibrary/MarkdownSidecar.java create mode 100644 jablib/src/test/java/org/jabref/logic/directorylibrary/MarkdownSidecarTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index fd5680c88155..1c778e38b3e2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv ### Added -- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear immediately and are enriched in the background with metadata extracted from the PDF itself and a generated citation key; a missing DOI is looked up online and its metadata fills the remaining empty fields. Directory libraries that were open on shutdown are reopened on the next start. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) +- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files, the Markdown notes files with a Hayagriva YAML frontmatter (`.md`; the notes body maps to the entry's comment fields), and the PDFs found in the folder tree. A PDF next to a sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear immediately and are enriched in the background with metadata extracted from the PDF itself and a generated citation key; a missing DOI is looked up online and its metadata fills the remaining empty fields. Directory libraries that were open on shutdown are reopened on the next start. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) - The HTTP import endpoint (`POST /libraries/{id}/entries`) now accepts CSL-JSON (`application/vnd.citationstyles.csl+json`), mapping each item to the correct entry type (e.g. conference paper, book chapter, thesis) via the citation-js-based mapping. [#16151](https://github.com/JabRef/jabref/pull/16151) - We added a new "Main" tab to the entry editor showing all fields of an entry in a single scrollable list, with one-click chips for adding optional fields and a free-form box for adding arbitrary fields. Identifiers, files and links, bibliometrics, comments, and meta fields (groups, owner, timestamps, special fields) live in collapsible sections — collapsed when empty — each offering chips for its unset fields. [#12711](https://github.com/JabRef/jabref/issues/12711) - We added auto-detection import for drag-and-dropped library files. [#15391](https://github.com/JabRef/jabref/issues/15391) diff --git a/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md b/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md index 1092e58d9248..5ec52d230f52 100644 --- a/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md +++ b/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md @@ -23,29 +23,43 @@ represented in JabRef, and where does the per-entry data live? ## Considered Options -* Hayagriva YAML sidecars (`X.yml` next to `X.pdf`), directory represented as a third - `DatabaseLocation` with an empty database path +* Hayagriva-based sidecars (Markdown notes files with a Hayagriva YAML frontmatter, plain + Hayagriva `.yml` also read), directory represented as a third `DatabaseLocation` with an + empty database path +* Pure Hayagriva YAML sidecars (`X.yml` next to `X.pdf`) * A hidden auto-maintained `.bib` file inside the directory * XMP metadata embedded in the PDFs as the only store ## Decision Outcome -Chosen option: "Hayagriva YAML sidecars with a third `DatabaseLocation`", because sidecars keep -the folder tool-agnostic (Hayagriva is Typst's bibliography format and JabRef has a symmetric -importer/exporter for it, including the `note` field), embedded XMP cannot represent all fields -and rewrites the PDFs themselves, and a hidden `.bib` would duplicate state that immediately -drifts from the files. +Chosen option: "Hayagriva-based sidecars with a third `DatabaseLocation`", because sidecars +keep the folder tool-agnostic (Hayagriva is Typst's bibliography format and JabRef has a +symmetric importer/exporter for it), embedded XMP cannot represent all fields and rewrites the +PDFs themselves, and a hidden `.bib` would duplicate state that immediately drifts from the +files. Pure YAML sidecars lost against the Markdown form because per-entry notes (JabRef's +comment fields) are long-form Markdown that reads terribly as YAML block scalars but naturally +as a Markdown body — the folder then doubles as a plain notes collection (Obsidian, any text +editor). Key points of the chosen design: +* A JabRef-authored sidecar is a Markdown file (`X.md` next to `X.pdf`): the YAML frontmatter + (between two `---` lines) is a regular Hayagriva document, the body below is markdownlint-clean + Markdown — a `# Notes` heading, the entry's comment text beneath it, and one + `## comment-` section per per-user comment field. Body content under other headings is + kept but not imported. Plain Hayagriva `.yml`/`.yaml` files are still read and written back + (they stay directly loadable by Typst); JabRef-only fields are written there as + `comment`/`comment-` extension keys, which the Hayagriva parser ignores. + * `DatabaseLocation.DIRECTORY`: the context keeps an **empty** database path plus a separate directory root. Empty path gives correct default behavior at almost every existing decision point (no autosave/backup managers, no `.bib` change monitor, "needs saved local database" actions disabled). The directory root is registered as the library-specific file directory, so relative PDF links resolve without a database path. -* Pairing is by convention — `X.yml`/`X.yaml` next to `X.pdf` — because Hayagriva has no +* Pairing is by convention — `X.md`/`X.yml`/`X.yaml` next to `X.pdf` — because Hayagriva has no file-path field; nothing JabRef-specific is written into the YAML for the association. -* Non-Hayagriva `.yml` files (CI configs, ...) are ignored via format recognition, not reported +* Non-Hayagriva `.yml` files (CI configs, ...) and `.md` files without a Hayagriva frontmatter + (READMEs, plain notes) are ignored via format recognition, not reported as errors. PDFs without a sidecar appear immediately as stubs; their metadata is extracted asynchronously after the library is shown (the standard PDF import pipeline, enriching the stub in place); a sidecar is only written once the user edits the entry (scanning never @@ -59,10 +73,13 @@ Key points of the chosen design: ### Consequences -* Good, because the directory stays the single source of truth and is usable from Typst as-is. +* Good, because the directory stays the single source of truth; `.yml` sidecars are usable from + Typst as-is, and `.md` sidecars double as plain Markdown notes. * Good, because the empty-database-path representation needs only a handful of explicit UI branches (tab title, close confirmation, save-as). +* Bad, because a `.md` sidecar is not directly loadable by Typst — its frontmatter must be + extracted (trivially, e.g. with `sed`/`yq`) or exported to obtain a plain Hayagriva file. * Bad, because YAML comments in hand-edited sidecars will not survive JabRef rewrites (the YAML - parser drops them), and JabRef-only fields need an extension mechanism inside the entry. + parser drops them). * Bad, because library-level metadata (groups, save actions) has no natural home yet; a metadata file in the root may be added later. diff --git a/docs/requirements/directory-library.md b/docs/requirements/directory-library.md index ffad2d8f861d..d663f8cb7121 100644 --- a/docs/requirements/directory-library.md +++ b/docs/requirements/directory-library.md @@ -4,15 +4,19 @@ parent: Requirements # Directory library ## Directory scan builds the library from sidecars and PDFs -`req~directory-library.scan~4` +`req~directory-library.scan~5` Opening a directory as a library must fill the library from the directory tree: every Hayagriva -`.yml`/`.yaml` file contributes its entries, a PDF with the same base name next to a sidecar is +`.yml`/`.yaml` file and every Markdown sidecar (`.md` opening with a Hayagriva YAML frontmatter +block; the notes body below maps to the entry's comment fields — the text under `# Notes` to +the comment, each `## comment-` section to that per-user comment) contributes its +entries, a PDF with the same base name next to a sidecar is linked to the sidecar's entry, and PDFs without a sidecar appear immediately as stub entries titled after the file; their metadata (embedded BibTeX, XMP, content heuristics — the standard PDF import pipeline) and a generated citation key arrive asynchronously after the library is shown, without replacing the entry instances. When the PDF yields no DOI, the DOI is looked up -online and the metadata behind it fills only the fields the PDF did not provide. Opening must not block on PDF parsing. Hidden files/directories, gitignored paths, and `.yml` files not recognized as Hayagriva +online and the metadata behind it fills only the fields the PDF did not provide. Opening must not block on PDF parsing. Hidden files/directories, gitignored paths, `.yml` files not recognized as Hayagriva, +and `.md` files without a Hayagriva frontmatter are skipped; unparseable Hayagriva files are reported as warnings without aborting the scan. Scanning must not write or modify any file in the directory. See [ADR 66](../decisions/0066-directory-as-library-with-hayagriva-sidecars.md) for more details. diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java index 1bb5c9472841..a6510ab2f50f 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java @@ -28,14 +28,15 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -/// Builds an in-memory library from a directory tree: each Hayagriva `.yml`/`.yaml` file -/// contributes its entries, a PDF next to a sidecar of the same base name is linked to the -/// sidecar's (first) entry, and PDFs without a sidecar become entries with metadata extracted -/// from the PDF itself (see [PdfEntryFactory]). The directory itself is the library — the -/// resulting [BibDatabaseContext] has [org.jabref.logic.shared.DatabaseLocation#DIRECTORY] and -/// no database path; linked files are stored relative to the root, which is registered as the +/// Builds an in-memory library from a directory tree: each Hayagriva `.yml`/`.yaml` file and +/// each Markdown sidecar (`.md` with Hayagriva frontmatter, see [MarkdownSidecar]) contributes +/// its entries, a PDF next to a sidecar of the same base name is linked to the sidecar's +/// (first) entry, and PDFs without a sidecar become entries with metadata extracted from the +/// PDF itself (see [PdfEntryFactory]). The directory itself is the library — the resulting +/// [BibDatabaseContext] has [org.jabref.logic.shared.DatabaseLocation#DIRECTORY] and no +/// database path; linked files are stored relative to the root, which is registered as the /// library-specific file directory. -// [impl->req~directory-library.scan~4] +// [impl->req~directory-library.scan~5] @NullMarked public class DirectoryLibraryScanner { @@ -57,6 +58,7 @@ public record PendingPdfImport(BibEntry entry, Path pdfFile) { private static final String PDF_EXTENSION = "pdf"; private final HayagrivaImporter importer = new HayagrivaImporter(); + private final MarkdownSidecar markdownSidecar = new MarkdownSidecar(); private final PdfEntryFactory pdfEntryFactory; public DirectoryLibraryScanner(PdfEntryFactory pdfEntryFactory) { @@ -72,26 +74,30 @@ public ScanResult scan(Path root) throws IOException { DirectoryLibraryCatalog catalog = new DirectoryLibraryCatalog(); List warnings = new ArrayList<>(); - List yamlFiles = new ArrayList<>(); + List sidecarFiles = new ArrayList<>(); List pdfFiles = new ArrayList<>(); - collectFiles(root, yamlFiles, pdfFiles); + collectFiles(root, sidecarFiles, pdfFiles); List entries = new ArrayList<>(); Set pairedPdfs = new HashSet<>(); - for (Path yamlFile : yamlFiles) { - // A directory may contain arbitrary YAML (CI configs, ...) — only files recognized - // as Hayagriva become entries; others are silently ignored - if (!looksLikeHayagriva(yamlFile)) { + for (Path sidecarFile : sidecarFiles) { + // A directory may contain arbitrary YAML (CI configs, ...) and arbitrary Markdown + // (READMEs, plain notes) — only files recognized as Hayagriva(-frontmatter) become + // entries; others are silently ignored + boolean isMarkdown = MarkdownSidecar.hasMarkdownExtension(sidecarFile); + if (isMarkdown ? !markdownSidecar.looksLikeSidecar(sidecarFile) : !looksLikeHayagriva(sidecarFile)) { continue; } - ParserResult parserResult = importer.importDatabase(yamlFile); + ParserResult parserResult = isMarkdown + ? markdownSidecar.read(sidecarFile) + : importer.importDatabase(sidecarFile); List fileEntries = parserResult.getDatabase().getEntries(); if (parserResult.isInvalid() || fileEntries.isEmpty()) { - warnings.add(Localization.lang("Could not parse the Hayagriva file '%0'.", yamlFile.toString())); + warnings.add(Localization.lang("Could not parse the Hayagriva file '%0'.", sidecarFile.toString())); continue; } - fileEntries.forEach(entry -> catalog.register(entry, yamlFile, entry.getCitationKey().orElse(""))); - findPairedPdf(yamlFile).ifPresent(pdf -> { + fileEntries.forEach(entry -> catalog.register(entry, sidecarFile, entry.getCitationKey().orElse(""))); + findPairedPdf(sidecarFile).ifPresent(pdf -> { pairedPdfs.add(pdf); linkPdf(fileEntries.getFirst(), root, pdf); }); @@ -114,7 +120,7 @@ public ScanResult scan(Path root) throws IOException { return new ScanResult(databaseContext, catalog, warnings, List.copyOf(pendingPdfImports)); } - private void collectFiles(Path root, List yamlFiles, List pdfFiles) throws IOException { + private void collectFiles(Path root, List sidecarFiles, List pdfFiles) throws IOException { GitIgnoreFileFilter gitIgnoreFilter = new GitIgnoreFileFilter(root); Files.walkFileTree(root, new SimpleFileVisitor<>() { @Override @@ -131,8 +137,8 @@ public FileVisitResult visitFile(Path file, BasicFileAttributes attributes) thro return FileVisitResult.CONTINUE; } String extension = FileUtil.getFileExtension(file).orElse(""); - if (YAML_EXTENSIONS.contains(extension)) { - yamlFiles.add(file); + if (YAML_EXTENSIONS.contains(extension) || MarkdownSidecar.MARKDOWN_EXTENSION.equals(extension)) { + sidecarFiles.add(file); } else if (PDF_EXTENSION.equals(extension)) { pdfFiles.add(file); } @@ -145,7 +151,7 @@ public FileVisitResult visitFileFailed(Path file, IOException exception) { return FileVisitResult.CONTINUE; } }); - yamlFiles.sort(Path::compareTo); + sidecarFiles.sort(Path::compareTo); pdfFiles.sort(Path::compareTo); } @@ -164,7 +170,7 @@ private boolean looksLikeHayagriva(Path yamlFile) throws IOException { } } - /// The sidecar convention: `X.yml` next to `X.pdf` (same directory, same base name). + /// The sidecar convention: `X.yml` (or `X.md`) next to `X.pdf` (same directory, same base name). private Optional findPairedPdf(Path yamlFile) { Path parent = yamlFile.getParent(); if (parent == null) { diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/MarkdownSidecar.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/MarkdownSidecar.java new file mode 100644 index 000000000000..293b3d132cd7 --- /dev/null +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/MarkdownSidecar.java @@ -0,0 +1,134 @@ +package org.jabref.logic.directorylibrary; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.Reader; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.Locale; +import java.util.Optional; + +import org.jabref.logic.importer.ParserResult; +import org.jabref.logic.importer.fileformat.HayagrivaImporter; +import org.jabref.logic.util.io.FileUtil; +import org.jabref.model.entry.BibEntry; +import org.jabref.model.entry.field.FieldFactory; +import org.jabref.model.entry.field.StandardField; + +import org.jspecify.annotations.NullMarked; + +/// A directory-library sidecar in Markdown form: `X.md` next to `X.pdf`. The YAML frontmatter +/// (between two `---` lines) is a regular Hayagriva document carrying the bibliographic data; +/// the Markdown body below carries JabRef's long-form notes. The text under the `# Notes` +/// heading is the entry's comment field, and every `## comment-` section is the +/// corresponding per-user comment field. Body content under other headings is not imported and +/// stays file-only, so the file remains a normal, markdownlint-clean Markdown note (usable in +/// Obsidian and plain editors); Typst users extract the frontmatter to obtain a plain Hayagriva +/// file. +/// +/// The body always describes the frontmatter's first entry; JabRef-authored Markdown sidecars +/// contain exactly one entry. +@NullMarked +public class MarkdownSidecar { + + public static final String MARKDOWN_EXTENSION = "md"; + static final String FRONTMATTER_DELIMITER = "---"; + + /// Field name (and name prefix) of JabRef's comment fields, see + /// [org.jabref.model.entry.field.UserSpecificCommentField]. + private static final String COMMENT_FIELD_PREFIX = "comment-"; + private static final String SECTION_HEADING_PREFIX = "## "; + + private final HayagrivaImporter importer = new HayagrivaImporter(); + + /// The split of a sidecar's raw text into its Hayagriva frontmatter and its notes body. + record Document(String frontmatter, String body) { + } + + public static boolean hasMarkdownExtension(Path file) { + return MARKDOWN_EXTENSION.equals(FileUtil.getFileExtension(file).orElse("").toLowerCase(Locale.ROOT)); + } + + /// Mirrors [HayagrivaImporter#isRecognizedFormat]: a Markdown file is a sidecar when it + /// opens with a frontmatter block that is recognized as Hayagriva. Arbitrary Markdown + /// (READMEs, plain notes) is no sidecar. + public boolean looksLikeSidecar(Path file) throws IOException { + Optional document = split(Files.readString(file, StandardCharsets.UTF_8)); + if (document.isEmpty()) { + return false; + } + try (BufferedReader reader = new BufferedReader(Reader.of(document.get().frontmatter()))) { + return importer.isRecognizedFormat(reader); + } + } + + /// Reads a Markdown sidecar: the frontmatter through the Hayagriva importer, the notes body + /// into the first entry's comment fields. A file without a frontmatter block yields an empty + /// result (callers recognize sidecars via [#looksLikeSidecar] first). + public ParserResult read(Path file) throws IOException { + Optional document = split(Files.readString(file, StandardCharsets.UTF_8)); + if (document.isEmpty()) { + return new ParserResult(); + } + ParserResult result; + try (BufferedReader reader = new BufferedReader(Reader.of(document.get().frontmatter()))) { + result = importer.importDatabase(reader); + } + List entries = result.getDatabase().getEntries(); + if (!entries.isEmpty()) { + applyBody(entries.getFirst(), document.get().body()); + } + return result; + } + + /// Splits the raw text at the frontmatter delimiters: the first line must be `---`, the + /// frontmatter runs until the next `---` line, the body is everything below. + static Optional split(String content) { + List lines = content.lines().toList(); + if (lines.isEmpty() || !FRONTMATTER_DELIMITER.equals(lines.getFirst().strip())) { + return Optional.empty(); + } + for (int end = 1; end < lines.size(); end++) { + if (FRONTMATTER_DELIMITER.equals(lines.get(end).strip())) { + return Optional.of(new Document( + String.join("\n", lines.subList(1, end)), + String.join("\n", lines.subList(end + 1, lines.size())))); + } + } + return Optional.empty(); + } + + /// The intro under the (optional) `# Notes` document heading becomes the comment field; + /// every `## comment-` section becomes the equally named per-user comment field. + static void applyBody(BibEntry entry, String body) { + String currentSection = ""; + boolean titleSkipped = false; + StringBuilder currentText = new StringBuilder(); + for (String line : body.lines().toList()) { + if (line.startsWith(SECTION_HEADING_PREFIX)) { + applySection(entry, currentSection, currentText.toString()); + currentSection = line.substring(SECTION_HEADING_PREFIX.length()).strip(); + currentText.setLength(0); + } else if (!titleSkipped && currentSection.isEmpty() && currentText.toString().isBlank() && line.startsWith("# ")) { + titleSkipped = true; + } else { + currentText.append(line).append('\n'); + } + } + applySection(entry, currentSection, currentText.toString()); + } + + private static void applySection(BibEntry entry, String section, String text) { + String value = text.strip(); + if (value.isEmpty()) { + return; + } + if (section.isEmpty()) { + entry.setField(StandardField.COMMENT, value); + } else if (section.startsWith(COMMENT_FIELD_PREFIX)) { + entry.setField(FieldFactory.parseField(section), value); + } + } +} diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java index fc6755d3072e..25dd9fc43001 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java @@ -18,7 +18,7 @@ /// /// Extraction runs on the background thread; entry mutations (and progress updates, which drive /// JavaFX properties) go through the injected marshaller. -// [impl->req~directory-library.scan~4] +// [impl->req~directory-library.scan~5] @NullMarked public class PdfEnrichmentTask extends BackgroundTask { diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java index 0cf3bef69d02..1e03bf99b143 100644 --- a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java @@ -21,6 +21,7 @@ import org.jabref.logic.shared.DatabaseLocation; import org.jabref.model.entry.BibEntry; import org.jabref.model.entry.LinkedFile; +import org.jabref.model.entry.field.FieldFactory; import org.jabref.model.entry.field.StandardField; import org.jabref.model.entry.types.StandardEntryType; @@ -111,6 +112,52 @@ void sidecarEntryIsImportedAndPairedWithPdf() throws IOException { result.catalog().sourceOf(entry)); } + @Test + void markdownSidecarIsImportedWithNotesAndPairedPdf() throws IOException { + Files.writeString(root.resolve("smith2020.md"), """ + --- + smith2020: + type: article + title: A Test Article + author: Smith, Jane + --- + + # Notes + + Shared comment text. + + ## comment-koppor + + Per-user comment text. + """); + Files.createFile(root.resolve("smith2020.pdf")); + + ScanResult result = scan(); + BibEntry entry = singleEntry(result); + + assertEquals(Optional.of("smith2020"), entry.getCitationKey()); + assertEquals(Optional.of("A Test Article"), entry.getField(StandardField.TITLE)); + assertEquals(Optional.of("Shared comment text."), entry.getField(StandardField.COMMENT)); + assertEquals(Optional.of("Per-user comment text."), entry.getField(FieldFactory.parseField("comment-koppor"))); + assertEquals(List.of(new LinkedFile("", Path.of("smith2020.pdf"), "PDF")), entry.getFiles()); + assertEquals(Optional.of(new DirectoryLibraryCatalog.EntrySource(root.resolve("smith2020.md"), "smith2020")), + result.catalog().sourceOf(entry)); + } + + @Test + void plainMarkdownFileIsIgnored() throws IOException { + Files.writeString(root.resolve("README.md"), """ + # A readme + + Just prose. + """); + + ScanResult result = scan(); + + assertEquals(List.of(), result.databaseContext().getDatabase().getEntries()); + assertEquals(List.of(), result.warnings()); + } + @Test void sidecarInNestedDirectoryLinksPdfRelativeToRoot() throws IOException { Path subDirectory = Files.createDirectories(root.resolve("conference").resolve("2020")); diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/MarkdownSidecarTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/MarkdownSidecarTest.java new file mode 100644 index 000000000000..0e82768ded6f --- /dev/null +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/MarkdownSidecarTest.java @@ -0,0 +1,152 @@ +package org.jabref.logic.directorylibrary; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.Optional; + +import org.jabref.model.entry.BibEntry; +import org.jabref.model.entry.field.FieldFactory; +import org.jabref.model.entry.field.StandardField; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class MarkdownSidecarTest { + + private static final String SIDECAR = """ + --- + smith2020: + type: article + title: A Test Article + author: Smith, Jane + --- + + # Notes + + Shared comment text. + + ## comment-koppor + + Per-user comment text. + + ## Ideas + + File-only content, not imported. + """; + + @TempDir + Path tempDir; + + private final MarkdownSidecar sidecar = new MarkdownSidecar(); + + private Path write(String fileName, String content) throws IOException { + Path file = tempDir.resolve(fileName); + Files.writeString(file, content); + return file; + } + + @Test + void readsFrontmatterEntryWithBodyComments() throws IOException { + Path file = write("smith2020.md", SIDECAR); + + List entries = sidecar.read(file).getDatabase().getEntries(); + + assertEquals(1, entries.size()); + BibEntry entry = entries.getFirst(); + assertEquals(Optional.of("smith2020"), entry.getCitationKey()); + assertEquals(Optional.of("A Test Article"), entry.getField(StandardField.TITLE)); + assertEquals(Optional.of("Shared comment text."), entry.getField(StandardField.COMMENT)); + assertEquals(Optional.of("Per-user comment text."), entry.getField(FieldFactory.parseField("comment-koppor"))); + } + + @Test + void recognizesSidecarByItsFrontmatter() throws IOException { + Path file = write("smith2020.md", SIDECAR); + + assertTrue(sidecar.looksLikeSidecar(file)); + } + + @Test + void plainMarkdownIsNoSidecar() throws IOException { + Path file = write("README.md", """ + # A readme + + Just prose. + """); + + assertFalse(sidecar.looksLikeSidecar(file)); + } + + @Test + void nonHayagrivaFrontmatterIsNoSidecar() throws IOException { + Path file = write("post.md", """ + --- + title: A blog post + layout: default + --- + + Content. + """); + + assertFalse(sidecar.looksLikeSidecar(file)); + } + + @Test + void bodyWithoutNotesHeadingStillBecomesComment() throws IOException { + Path file = write("smith2020.md", """ + --- + smith2020: + type: article + title: A Test Article + --- + + Comment without a heading. + """); + + BibEntry entry = sidecar.read(file).getDatabase().getEntries().getFirst(); + + assertEquals(Optional.of("Comment without a heading."), entry.getField(StandardField.COMMENT)); + } + + @Test + void multiParagraphCommentKeepsItsInnerBlankLine() throws IOException { + Path file = write("smith2020.md", """ + --- + smith2020: + type: article + title: A Test Article + --- + + # Notes + + First paragraph. + + Second paragraph. + """); + + BibEntry entry = sidecar.read(file).getDatabase().getEntries().getFirst(); + + assertEquals(Optional.of("First paragraph.\n\nSecond paragraph."), entry.getField(StandardField.COMMENT)); + } + + @Test + void emptyBodySetsNoCommentFields() throws IOException { + Path file = write("smith2020.md", """ + --- + smith2020: + type: article + title: A Test Article + --- + """); + + BibEntry entry = sidecar.read(file).getDatabase().getEntries().getFirst(); + + assertEquals(Optional.empty(), entry.getField(StandardField.COMMENT)); + } +} From ae1aee608f64a4512acf8950f242dbeb0ed42a78 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Thu, 3 Sep 2026 03:38:38 +0200 Subject: [PATCH 09/16] Renumber directory-library ADR to 0071 Upstream took 0066 in the meantime; the MADR check rejects the duplicate id. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01Vr3E1Gg5DRU4LQDDVnhPys --- ...rs.md => 0071-directory-as-library-with-hayagriva-sidecars.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/decisions/{0066-directory-as-library-with-hayagriva-sidecars.md => 0071-directory-as-library-with-hayagriva-sidecars.md} (100%) diff --git a/docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md b/docs/decisions/0071-directory-as-library-with-hayagriva-sidecars.md similarity index 100% rename from docs/decisions/0066-directory-as-library-with-hayagriva-sidecars.md rename to docs/decisions/0071-directory-as-library-with-hayagriva-sidecars.md From 8e59a93bd545907c4cb8bb81fd3f382f6c5960c8 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Thu, 3 Sep 2026 04:19:45 +0200 Subject: [PATCH 10/16] Harden the directory-library opener A failed "Save as" no longer strips the directory library of its root; the background PDF enrichment is cancelled with the tab, logs failures, and no longer marks the library as modified; opening an already open folder raises its tab; extensions match case-insensitively and gitignored PDFs are not paired. Optional/null idioms follow the checklist. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01Vr3E1Gg5DRU4LQDDVnhPys --- CHANGELOG.md | 2 +- ...tory-as-library-with-hayagriva-sidecars.md | 2 +- docs/requirements/directory-library.md | 2 +- .../main/java/org/jabref/gui/LibraryTab.java | 11 +++- .../gui/exporter/SaveDatabaseAction.java | 8 +-- .../gui/frame/JabRefFrameViewModel.java | 4 +- .../importer/actions/OpenDatabaseAction.java | 7 +-- .../actions/OpenDirectoryLibraryAction.java | 53 ++++++++++++++----- .../DirectoryLibraryScanner.java | 35 +++++------- .../directorylibrary/MarkdownSidecar.java | 34 ++++++++---- .../directorylibrary/PdfEnrichmentTask.java | 11 ++-- .../directorylibrary/PdfEntryFactory.java | 26 +++++---- .../model/database/BibDatabaseContext.java | 6 +++ .../DirectoryLibraryScannerTest.java | 31 +++++++++-- 14 files changed, 155 insertions(+), 77 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 53796afbb588..3a6c36f00856 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv ### Added -- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files, the Markdown notes files with a Hayagriva YAML frontmatter (`.md`; the notes body maps to the entry's comment fields), and the PDFs found in the folder tree. A PDF next to a sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear immediately and are enriched in the background with metadata extracted from the PDF itself and a generated citation key; a missing DOI is looked up online and its metadata fills the remaining empty fields. Directory libraries that were open on shutdown are reopened on the next start. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) +- We added "Open folder as library" (File menu): a folder of PDFs and Hayagriva sidecar files (`.yml`, or `.md` notes with a Hayagriva frontmatter) opens as a library, and it is reopened on the next start. PDFs without a sidecar appear right away and get their metadata extracted in the background. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) - We added a "Commit and push" button which allows to commit and then push in one go for Git operations. [#16339](https://github.com/JabRef/jabref/issues/16339) - We added the option to close and reopen the PDF preview in the unlinked files dialog. [#16159](https://github.com/JabRef/jabref/issues/16159) - We added the ability for LibreOffice BST citations to use style-defined labels. [forum#3764]([https://github.com/JabRef/jabref/issues/16357](https://discourse.jabref.org/t/feature-request-custom-citation-styles-from-bst/3764)) diff --git a/docs/decisions/0071-directory-as-library-with-hayagriva-sidecars.md b/docs/decisions/0071-directory-as-library-with-hayagriva-sidecars.md index 5ec52d230f52..5229eadcaf1f 100644 --- a/docs/decisions/0071-directory-as-library-with-hayagriva-sidecars.md +++ b/docs/decisions/0071-directory-as-library-with-hayagriva-sidecars.md @@ -1,5 +1,5 @@ --- -nav_order: 0066 +nav_order: 0071 parent: Decision Records --- diff --git a/docs/requirements/directory-library.md b/docs/requirements/directory-library.md index d663f8cb7121..7a54c2ae86d7 100644 --- a/docs/requirements/directory-library.md +++ b/docs/requirements/directory-library.md @@ -19,7 +19,7 @@ online and the metadata behind it fills only the fields the PDF did not provide. and `.md` files without a Hayagriva frontmatter are skipped; unparseable Hayagriva files are reported as warnings without aborting the scan. Scanning must not write or modify any file in the directory. -See [ADR 66](../decisions/0066-directory-as-library-with-hayagriva-sidecars.md) for more details. +See [ADR 71](../decisions/0071-directory-as-library-with-hayagriva-sidecars.md) for more details. Needs: impl diff --git a/jabgui/src/main/java/org/jabref/gui/LibraryTab.java b/jabgui/src/main/java/org/jabref/gui/LibraryTab.java index 78dd26205b24..f63b778a8a42 100644 --- a/jabgui/src/main/java/org/jabref/gui/LibraryTab.java +++ b/jabgui/src/main/java/org/jabref/gui/LibraryTab.java @@ -83,6 +83,7 @@ import org.jabref.model.entry.BibEntryTypesManager; import org.jabref.model.entry.BibtexString; import org.jabref.model.entry.LinkedFile; +import org.jabref.model.entry.event.EntriesEvent; import org.jabref.model.entry.event.EntriesEventSource; import org.jabref.model.entry.event.FieldChangedEvent; import org.jabref.model.entry.field.FieldFactory; @@ -484,6 +485,13 @@ public void updateTabTitle(boolean isChanged) { @Subscribe public void listen(BibDatabaseContextChangedEvent event) { + // Background enrichment of a directory library is system-initiated (SHARED-sourced), + // not something the user would be asked to save + if (bibDatabaseContext.getLocation() == DatabaseLocation.DIRECTORY + && event instanceof EntriesEvent entriesEvent + && entriesEvent.getEntriesEventSource() == EntriesEventSource.SHARED) { + return; + } this.changedProperty.setValue(true); } @@ -657,8 +665,7 @@ private boolean confirmClose() { } String filename = getBibDatabaseContext() - .getDatabasePath() - .or(() -> getBibDatabaseContext().getDirectoryLibraryRoot()) + .getPathOnDisk() .map(Path::toAbsolutePath) .map(Path::toString) .orElse(Localization.lang("untitled")); diff --git a/jabgui/src/main/java/org/jabref/gui/exporter/SaveDatabaseAction.java b/jabgui/src/main/java/org/jabref/gui/exporter/SaveDatabaseAction.java index 98b48c18ae21..805629523cf2 100644 --- a/jabgui/src/main/java/org/jabref/gui/exporter/SaveDatabaseAction.java +++ b/jabgui/src/main/java/org/jabref/gui/exporter/SaveDatabaseAction.java @@ -158,9 +158,6 @@ boolean saveAs(Path file, SaveDatabaseMode mode) { // Save all properties dependent on the ID. This makes it possible to restore them. new SharedDatabasePreferences(context.getDatabase().generateSharedDatabaseID()) .putAllDBMSConnectionProperties(context.getDBMSSynchronizer().getConnectionProperties()); - } else if (context.getLocation() == DatabaseLocation.DIRECTORY) { - // "Save as" snapshots a directory library into a regular .bib library - context.convertToLocalDatabase(); } SaveResult saveResult = save(file, mode); @@ -172,6 +169,11 @@ boolean saveAs(Path file, SaveDatabaseMode mode) { if (saveResult == SaveResult.SUCCESS) { // we managed to successfully save the file // thus, we can store the path into the context + if (context.getLocation() == DatabaseLocation.DIRECTORY) { + // "Save as" snapshots a directory library into a regular .bib library; only now, + // so a failed save leaves the directory library intact + context.convertToLocalDatabase(); + } context.setDatabasePath(file); stateManager.setActiveDatabase(context); libraryTab.updateTabTitle(false); diff --git a/jabgui/src/main/java/org/jabref/gui/frame/JabRefFrameViewModel.java b/jabgui/src/main/java/org/jabref/gui/frame/JabRefFrameViewModel.java index 416887097245..d6e4e5e94350 100644 --- a/jabgui/src/main/java/org/jabref/gui/frame/JabRefFrameViewModel.java +++ b/jabgui/src/main/java/org/jabref/gui/frame/JabRefFrameViewModel.java @@ -133,13 +133,13 @@ public boolean close() { // [impl->req~directory-library.session-restore~1] List openedLibraries = tabContainer.getLibraryTabs().stream() .map(LibraryTab::getBibDatabaseContext) - .map(context -> context.getDatabasePath().or(context::getDirectoryLibraryRoot)) + .map(BibDatabaseContext::getPathOnDisk) .flatMap(Optional::stream) .map(Path::toAbsolutePath) .toList(); Path focusedLibraries = Optional.ofNullable(tabContainer.getCurrentLibraryTab()) .map(LibraryTab::getBibDatabaseContext) - .flatMap(context -> context.getDatabasePath().or(context::getDirectoryLibraryRoot)) + .flatMap(BibDatabaseContext::getPathOnDisk) .map(Path::toAbsolutePath) .orElse(null); diff --git a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDatabaseAction.java b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDatabaseAction.java index 5184e8e12396..de7bcf8857b0 100644 --- a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDatabaseAction.java +++ b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDatabaseAction.java @@ -165,13 +165,14 @@ public void openFile(Path file) { public void openFiles(List filesToOpen) { // Directories are directory libraries (e.g. restored from the last session) // [impl->req~directory-library.session-restore~1] + OpenDirectoryLibraryAction openDirectoryLibraryAction = new OpenDirectoryLibraryAction(tabContainer, dialogService, preferences, + aiService, stateManager, fileUpdateMonitor, entryTypesManager, undoManager, clipboardManager, taskExecutor); filesToOpen.stream() .map(FileUtil::resolveIfShortcut) .filter(Files::isDirectory) + .map(directory -> directory.toAbsolutePath().normalize()) .distinct() - .forEach(directory -> new OpenDirectoryLibraryAction(tabContainer, dialogService, preferences, - aiService, stateManager, fileUpdateMonitor, entryTypesManager, undoManager, - clipboardManager, taskExecutor).openDirectory(directory)); + .forEach(openDirectoryLibraryAction::openDirectory); // Resolve any shortcuts to their targets and filter to only .bib files. // The resulting list must remain modifiable for downstream processing (iterator.remove() calls below). diff --git a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java index 28ff3f0ba4bf..3880cc29ee96 100644 --- a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java +++ b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java @@ -2,6 +2,9 @@ import java.nio.file.Path; +import javafx.event.Event; +import javafx.event.EventHandler; + import org.jabref.gui.DialogService; import org.jabref.gui.LibraryTab; import org.jabref.gui.LibraryTabContainer; @@ -22,10 +25,15 @@ import org.jabref.model.entry.BibEntryTypesManager; import org.jabref.model.util.FileUpdateMonitor; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + /// Opens a directory as a library: the main table fills from the Hayagriva `.yml` sidecars and /// `.pdf` files found in the directory tree (see [DirectoryLibraryScanner]). public class OpenDirectoryLibraryAction extends SimpleCommand { + private static final Logger LOGGER = LoggerFactory.getLogger(OpenDirectoryLibraryAction.class); + private final LibraryTabContainer tabContainer; private final DialogService dialogService; private final GuiPreferences preferences; @@ -65,18 +73,31 @@ public void execute() { .withInitialDirectory(preferences.getFilePreferences().getWorkingDirectory()) .build(); dialogService.showDirectorySelectionDialog(directoryDialogConfiguration) - .ifPresent(this::openDirectory); + .ifPresent(root -> { + preferences.getFilePreferences().setWorkingDirectory(root); + openDirectory(root); + }); } /// Opens the directory as a library without showing the chooser (used by the session - /// restore and internal routing). - public void openDirectory(Path root) { - preferences.getFilePreferences().setWorkingDirectory(root); + /// restore and internal routing). An already open directory library is raised instead. + public void openDirectory(Path directory) { + Path root = directory.toAbsolutePath().normalize(); + tabContainer.getLibraryTabs().stream() + .filter(tab -> tab.getBibDatabaseContext().getDirectoryLibraryRoot() + .map(openRoot -> openRoot.toAbsolutePath().normalize()) + .filter(root::equals) + .isPresent()) + .findFirst() + .ifPresentOrElse(tabContainer::showLibraryTab, () -> scanAndShow(root)); + } + + private void scanAndShow(Path root) { PdfEntryFactory pdfEntryFactory = new PdfEntryFactory( preferences.getImportFormatPreferences(), preferences.getFilePreferences(), preferences.getCitationKeyPatternPreferences()); BackgroundTask.wrap(() -> new DirectoryLibraryScanner(pdfEntryFactory).scan(root)) - .onSuccess(this::showLibraryTab) + .onSuccess(scanResult -> showLibraryTab(scanResult, pdfEntryFactory)) .onFailure(exception -> dialogService.showErrorDialogAndWait( Localization.lang("Open folder as library"), Localization.lang("Could not open folder '%0' as library.", root.toString()), @@ -84,7 +105,7 @@ public void openDirectory(Path root) { .executeWith(taskExecutor); } - private void showLibraryTab(DirectoryLibraryScanner.ScanResult scanResult) { + private void showLibraryTab(DirectoryLibraryScanner.ScanResult scanResult, PdfEntryFactory pdfEntryFactory) { // The synchronous factory keeps the DIRECTORY location: the ParserResult-based one // reconstructs a fresh (LOCAL) context from database + metadata on loading success LibraryTab libraryTab = LibraryTab.createLibraryTab( @@ -101,12 +122,11 @@ private void showLibraryTab(DirectoryLibraryScanner.ScanResult scanResult) { taskExecutor); tabContainer.addTab(libraryTab, true); if (!scanResult.pendingPdfImports().isEmpty()) { - PdfEntryFactory pdfEntryFactory = new PdfEntryFactory( - preferences.getImportFormatPreferences(), preferences.getFilePreferences(), - preferences.getCitationKeyPatternPreferences()); - new PdfEnrichmentTask(scanResult.pendingPdfImports(), pdfEntryFactory, scanResult.databaseContext(), - UiTaskExecutor::runInJavaFXThread) - .executeWith(taskExecutor); + PdfEnrichmentTask enrichment = new PdfEnrichmentTask(scanResult.pendingPdfImports(), pdfEntryFactory, + scanResult.databaseContext(), UiTaskExecutor::runInJavaFXThread); + enrichment.onFailure(exception -> LOGGER.error("Extracting PDF metadata failed", exception)); + cancelOnClose(libraryTab, enrichment); + enrichment.executeWith(taskExecutor); } if (!scanResult.warnings().isEmpty()) { @@ -115,4 +135,13 @@ private void showLibraryTab(DirectoryLibraryScanner.ScanResult scanResult) { String.join("\n", scanResult.warnings())); } } + + /// The enrichment mutates entries of the tab's library, so it must not outlive the tab. + private static void cancelOnClose(LibraryTab libraryTab, PdfEnrichmentTask enrichment) { + EventHandler onClosed = libraryTab.getOnClosed(); + libraryTab.setOnClosed(event -> { + enrichment.cancel(); + onClosed.handle(event); + }); + } } diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java index a6510ab2f50f..b89659528e68 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java @@ -9,8 +9,11 @@ import java.nio.file.SimpleFileVisitor; import java.nio.file.attribute.BasicFileAttributes; import java.util.ArrayList; +import java.util.HashMap; import java.util.HashSet; import java.util.List; +import java.util.Locale; +import java.util.Map; import java.util.Optional; import java.util.Set; @@ -77,6 +80,10 @@ public ScanResult scan(Path root) throws IOException { List sidecarFiles = new ArrayList<>(); List pdfFiles = new ArrayList<>(); collectFiles(root, sidecarFiles, pdfFiles); + // The sidecar convention: `X.yml` (or `X.md`) next to `X.pdf` (same directory, same + // base name); only PDFs that passed the skip rules above can be paired + Map pdfByStem = new HashMap<>(); + pdfFiles.forEach(pdf -> pdfByStem.put(pdf.resolveSibling(FileUtil.getBaseName(pdf)), pdf)); List entries = new ArrayList<>(); Set pairedPdfs = new HashSet<>(); @@ -96,8 +103,8 @@ public ScanResult scan(Path root) throws IOException { warnings.add(Localization.lang("Could not parse the Hayagriva file '%0'.", sidecarFile.toString())); continue; } - fileEntries.forEach(entry -> catalog.register(entry, sidecarFile, entry.getCitationKey().orElse(""))); - findPairedPdf(sidecarFile).ifPresent(pdf -> { + fileEntries.forEach(entry -> catalog.register(entry, sidecarFile, entry.getCitationKey().orElseThrow())); + Optional.ofNullable(pdfByStem.get(sidecarFile.resolveSibling(FileUtil.getBaseName(sidecarFile)))).ifPresent(pdf -> { pairedPdfs.add(pdf); linkPdf(fileEntries.getFirst(), root, pdf); }); @@ -136,7 +143,7 @@ public FileVisitResult visitFile(Path file, BasicFileAttributes attributes) thro if (isHidden(file) || !gitIgnoreFilter.accept(file)) { return FileVisitResult.CONTINUE; } - String extension = FileUtil.getFileExtension(file).orElse(""); + String extension = FileUtil.getFileExtension(file).map(ext -> ext.toLowerCase(Locale.ROOT)).orElse(""); if (YAML_EXTENSIONS.contains(extension) || MarkdownSidecar.MARKDOWN_EXTENSION.equals(extension)) { sidecarFiles.add(file); } else if (PDF_EXTENSION.equals(extension)) { @@ -156,8 +163,10 @@ public FileVisitResult visitFileFailed(Path file, IOException exception) { } private static boolean isHidden(Path path) { - Path fileName = path.getFileName(); - return fileName != null && fileName.toString().startsWith("."); + return Optional.ofNullable(path.getFileName()) + .map(Path::toString) + .filter(name -> name.startsWith(".")) + .isPresent(); } /// Uses the lookahead-based recognition (a `type:` line naming a Hayagriva entry type) @@ -170,22 +179,6 @@ private boolean looksLikeHayagriva(Path yamlFile) throws IOException { } } - /// The sidecar convention: `X.yml` (or `X.md`) next to `X.pdf` (same directory, same base name). - private Optional findPairedPdf(Path yamlFile) { - Path parent = yamlFile.getParent(); - if (parent == null) { - return Optional.empty(); - } - String baseName = FileUtil.getBaseName(yamlFile); - for (String candidate : List.of(baseName + ".pdf", baseName + ".PDF")) { - Path pdf = parent.resolve(candidate); - if (Files.exists(pdf)) { - return Optional.of(pdf); - } - } - return Optional.empty(); - } - private void linkPdf(BibEntry entry, Path root, Path pdf) { if (entry.getFiles().isEmpty()) { entry.addFile(new LinkedFile("", root.relativize(pdf), StandardFileType.PDF.getName())); diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/MarkdownSidecar.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/MarkdownSidecar.java index 293b3d132cd7..971b65d1cfe1 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/MarkdownSidecar.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/MarkdownSidecar.java @@ -3,6 +3,7 @@ import java.io.BufferedReader; import java.io.IOException; import java.io.Reader; +import java.io.UncheckedIOException; import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; @@ -55,12 +56,18 @@ public static boolean hasMarkdownExtension(Path file) { /// opens with a frontmatter block that is recognized as Hayagriva. Arbitrary Markdown /// (READMEs, plain notes) is no sidecar. public boolean looksLikeSidecar(Path file) throws IOException { - Optional document = split(Files.readString(file, StandardCharsets.UTF_8)); - if (document.isEmpty()) { - return false; - } - try (BufferedReader reader = new BufferedReader(Reader.of(document.get().frontmatter()))) { + return split(Files.readString(file, StandardCharsets.UTF_8)) + .map(Document::frontmatter) + .map(this::isHayagrivaFrontmatter) + .orElse(false); + } + + private boolean isHayagrivaFrontmatter(String frontmatter) { + try (BufferedReader reader = new BufferedReader(Reader.of(frontmatter))) { return importer.isRecognizedFormat(reader); + } catch (IOException e) { + // Reading from an in-memory string cannot fail + throw new UncheckedIOException(e); } } @@ -68,17 +75,22 @@ public boolean looksLikeSidecar(Path file) throws IOException { /// into the first entry's comment fields. A file without a frontmatter block yields an empty /// result (callers recognize sidecars via [#looksLikeSidecar] first). public ParserResult read(Path file) throws IOException { - Optional document = split(Files.readString(file, StandardCharsets.UTF_8)); - if (document.isEmpty()) { - return new ParserResult(); - } + return split(Files.readString(file, StandardCharsets.UTF_8)) + .map(this::read) + .orElseGet(ParserResult::new); + } + + private ParserResult read(Document document) { ParserResult result; - try (BufferedReader reader = new BufferedReader(Reader.of(document.get().frontmatter()))) { + try (BufferedReader reader = new BufferedReader(Reader.of(document.frontmatter()))) { result = importer.importDatabase(reader); + } catch (IOException e) { + // Reading from an in-memory string cannot fail + throw new UncheckedIOException(e); } List entries = result.getDatabase().getEntries(); if (!entries.isEmpty()) { - applyBody(entries.getFirst(), document.get().body()); + applyBody(entries.getFirst(), document.body()); } return result; } diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java index 25dd9fc43001..89be73aaf67d 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java @@ -16,8 +16,8 @@ /// fills progressively and opening the folder never blocks on PDF parsing or identifier /// lookups. Cancelling keeps the stubs created by the scan. /// -/// Extraction runs on the background thread; entry mutations (and progress updates, which drive -/// JavaFX properties) go through the injected marshaller. +/// Extraction runs on the background thread; entry mutations go through the injected +/// marshaller (the JavaFX thread in the GUI). // [impl->req~directory-library.scan~5] @NullMarked public class PdfEnrichmentTask extends BackgroundTask { @@ -47,11 +47,8 @@ public Void call() { break; } counter++; - final int progress = counter; - modelUpdateMarshaller.accept(() -> { - updateMessage(pending.pdfFile().getFileName().toString()); - updateProgress(progress, pendingImports.size()); - }); + updateMessage(pending.pdfFile().getFileName().toString()); + updateProgress(counter, pendingImports.size()); Optional extracted = pdfEntryFactory.extractMetadata(pending.pdfFile(), databaseContext); modelUpdateMarshaller.accept(() -> { diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java index 391456b39d1a..7e4ebe160a2a 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEntryFactory.java @@ -88,10 +88,9 @@ public BibEntry createEntry(Path pdf, Path root, BibDatabaseContext databaseCont /// The immediately available placeholder for a PDF: title from the file name, PDF linked. /// Metadata extraction happens asynchronously afterwards (see [PdfEnrichmentTask]). public BibEntry createStub(Path pdf, Path root) { - BibEntry stub = new BibEntry(StandardEntryType.Misc) - .withField(StandardField.TITLE, FileUtil.getBaseName(pdf)); - stub.addFile(new LinkedFile("", root.relativize(pdf), StandardFileType.PDF.getName())); - return stub; + return new BibEntry(StandardEntryType.Misc) + .withField(StandardField.TITLE, FileUtil.getBaseName(pdf)) + .withFiles(List.of(new LinkedFile("", root.relativize(pdf), StandardFileType.PDF.getName()))); } /// Applies extracted metadata onto the live stub entry (same instance, so table selection @@ -133,14 +132,21 @@ private void completeWithMissingDoiMetadata(BibEntry entry) { return; } try { - Optional doi = crossRef.findIdentifier(entry); - if (doi.isEmpty()) { - return; - } - entry.setField(StandardField.DOI, doi.get().asString()); - doiFetcher.performSearchById(doi.get().asString()).ifPresent(entry::mergeWith); + crossRef.findIdentifier(entry).map(DOI::asString).ifPresent(doi -> { + entry.setField(StandardField.DOI, doi); + fetchByDoi(doi).ifPresent(entry::mergeWith); + }); } catch (FetcherException e) { LOGGER.debug("DOI lookup failed for {}", entry.getAuthorTitleYear(80), e); } } + + private Optional fetchByDoi(String doi) { + try { + return doiFetcher.performSearchById(doi); + } catch (FetcherException e) { + LOGGER.debug("Could not fetch metadata for DOI {}", doi, e); + return Optional.empty(); + } + } } diff --git a/jablib/src/main/java/org/jabref/model/database/BibDatabaseContext.java b/jablib/src/main/java/org/jabref/model/database/BibDatabaseContext.java index a53a63c38cd8..c23ea0616547 100644 --- a/jablib/src/main/java/org/jabref/model/database/BibDatabaseContext.java +++ b/jablib/src/main/java/org/jabref/model/database/BibDatabaseContext.java @@ -283,6 +283,12 @@ public Optional getDirectoryLibraryRoot() { return Optional.ofNullable(directoryLibraryRoot); } + /// The place this library lives at on disk: the `.bib` file, or the root of a directory + /// library. Empty for unsaved and shared libraries. + public Optional getPathOnDisk() { + return getDatabasePath().or(this::getDirectoryLibraryRoot); + } + public void convertToLocalDatabase() { if (dbmsListener != null && (location == DatabaseLocation.SHARED)) { if (dbmsSynchronizer != null) { diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java index 1e03bf99b143..b5eb2113ad7c 100644 --- a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java @@ -18,6 +18,7 @@ import org.jabref.logic.importer.fetcher.CrossRef; import org.jabref.logic.importer.fetcher.DoiFetcher; import org.jabref.logic.importer.util.GrobidPreferences; +import org.jabref.logic.l10n.Localization; import org.jabref.logic.shared.DatabaseLocation; import org.jabref.model.entry.BibEntry; import org.jabref.model.entry.LinkedFile; @@ -31,7 +32,6 @@ import static org.jabref.logic.citationkeypattern.CitationKeyPatternPreferences.DEFAULT_UNWANTED_CHARACTERS; import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; @@ -256,8 +256,33 @@ void unparseableHayagrivaFileIsReportedAsWarning() throws IOException { ScanResult result = scan(); assertEquals(List.of(), result.databaseContext().getDatabase().getEntries()); - assertEquals(1, result.warnings().size()); - assertTrue(result.warnings().getFirst().contains("broken.yml")); + assertEquals(List.of(Localization.lang("Could not parse the Hayagriva file '%0'.", root.resolve("broken.yml").toString())), + result.warnings()); + } + + @Test + void extensionsAreMatchedCaseInsensitively() throws IOException { + Files.writeString(root.resolve("smith2020.YML"), ARTICLE_YAML); + Files.createFile(root.resolve("smith2020.PDF")); + Files.createFile(root.resolve("other.Pdf")); + + ScanResult result = scan(); + + List entries = result.databaseContext().getDatabase().getEntries(); + assertEquals(List.of(new LinkedFile("", Path.of("smith2020.PDF"), "PDF")), entries.getFirst().getFiles()); + assertEquals(List.of(new LinkedFile("", Path.of("other.Pdf"), "PDF")), entries.getLast().getFiles()); + assertEquals(2, entries.size()); + } + + @Test + void gitignoredPdfIsNotPairedWithSidecar() throws IOException { + Files.writeString(root.resolve(".gitignore"), "*.pdf\n"); + Files.writeString(root.resolve("smith2020.yml"), ARTICLE_YAML); + Files.createFile(root.resolve("smith2020.pdf")); + + ScanResult result = scan(); + + assertEquals(List.of(), singleEntry(result).getFiles()); } @Test From e5ddc86cd224c45a8a5d220905b10c6c33a48dc1 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Mon, 7 Sep 2026 00:15:38 +0200 Subject: [PATCH 11/16] Apply IntelliJ formatter after upstream resync Fixes the CI format check. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DDcHNMt9fPWnpYaHheFvry --- .../org/jabref/logic/exporter/HayagrivaEntryWriter.java | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/jablib/src/main/java/org/jabref/logic/exporter/HayagrivaEntryWriter.java b/jablib/src/main/java/org/jabref/logic/exporter/HayagrivaEntryWriter.java index 46f439a2fd7f..271ac32dd295 100644 --- a/jablib/src/main/java/org/jabref/logic/exporter/HayagrivaEntryWriter.java +++ b/jablib/src/main/java/org/jabref/logic/exporter/HayagrivaEntryWriter.java @@ -313,9 +313,12 @@ private ObjectNode parentNode(String type, String title) { private String booktitleParentType(EntryType entryType) { return switch (entryType) { - case StandardEntryType.InProceedings -> "proceedings"; - case StandardEntryType.InBook -> "book"; - default -> "anthology"; + case StandardEntryType.InProceedings -> + "proceedings"; + case StandardEntryType.InBook -> + "book"; + default -> + "anthology"; }; } From 0d090d09b76c99328ed17c8feaf53e974f087dc1 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Mon, 7 Sep 2026 10:46:42 +0200 Subject: [PATCH 12/16] Merge hayagriva-writer; adapt to createLibraryTab signature change Upstream dropped the undo manager from LibraryTab.createLibraryTab and added a GitHandlerRegistry; OpenDirectoryLibraryAction and its callers follow suit. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DDcHNMt9fPWnpYaHheFvry --- .../src/main/java/org/jabref/gui/frame/MainMenu.java | 2 +- .../gui/importer/actions/OpenDatabaseAction.java | 2 +- .../importer/actions/OpenDirectoryLibraryAction.java | 12 ++++++------ 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/jabgui/src/main/java/org/jabref/gui/frame/MainMenu.java b/jabgui/src/main/java/org/jabref/gui/frame/MainMenu.java index 9a38b5caa503..bfba51f6b952 100644 --- a/jabgui/src/main/java/org/jabref/gui/frame/MainMenu.java +++ b/jabgui/src/main/java/org/jabref/gui/frame/MainMenu.java @@ -169,7 +169,7 @@ private void createMenu() { file.getItems().addAll( factory.createMenuItem(StandardActions.NEW_LIBRARY, new NewDatabaseAction(frame, preferences)), factory.createMenuItem(StandardActions.OPEN_LIBRARY, openDatabaseActionSupplier.get()), - factory.createMenuItem(StandardActions.OPEN_FOLDER_AS_LIBRARY, new OpenDirectoryLibraryAction(frame, dialogService, preferences, aiService, stateManager, fileUpdateMonitor, entryTypesManager, undoManager, clipBoardManager, taskExecutor)), + factory.createMenuItem(StandardActions.OPEN_FOLDER_AS_LIBRARY, new OpenDirectoryLibraryAction(frame, dialogService, preferences, aiService, stateManager, fileUpdateMonitor, entryTypesManager, gitHandlerRegistry, clipBoardManager, taskExecutor)), fileHistoryMenu, factory.createMenuItem(StandardActions.SAVE_LIBRARY, new SaveAction(SaveAction.SaveMethod.SAVE, frame::getCurrentLibraryTab, dialogService, preferences, stateManager, entryTypesManager, journalAbbreviationRepository)), factory.createMenuItem(StandardActions.SAVE_LIBRARY_AS, new SaveAction(SaveAction.SaveMethod.SAVE_AS, frame::getCurrentLibraryTab, dialogService, preferences, stateManager, entryTypesManager, journalAbbreviationRepository)), diff --git a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDatabaseAction.java b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDatabaseAction.java index 0db85a4a1131..9ada32cd3d47 100644 --- a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDatabaseAction.java +++ b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDatabaseAction.java @@ -166,7 +166,7 @@ public void openFiles(List filesToOpen) { // Directories are directory libraries (e.g. restored from the last session) // [impl->req~directory-library.session-restore~1] OpenDirectoryLibraryAction openDirectoryLibraryAction = new OpenDirectoryLibraryAction(tabContainer, dialogService, preferences, - aiService, stateManager, fileUpdateMonitor, entryTypesManager, undoManager, clipboardManager, taskExecutor); + aiService, stateManager, fileUpdateMonitor, entryTypesManager, gitHandlerRegistry, clipboardManager, taskExecutor); filesToOpen.stream() .map(FileUtil::resolveIfShortcut) .filter(Files::isDirectory) diff --git a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java index 3880cc29ee96..d0fc7e2fc1f0 100644 --- a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java +++ b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java @@ -12,13 +12,13 @@ import org.jabref.gui.actions.SimpleCommand; import org.jabref.gui.clipboard.ClipBoardManager; import org.jabref.gui.preferences.GuiPreferences; -import org.jabref.gui.undo.GuiUndoManager; import org.jabref.gui.util.DirectoryDialogConfiguration; import org.jabref.gui.util.UiTaskExecutor; import org.jabref.logic.ai.AiService; import org.jabref.logic.directorylibrary.DirectoryLibraryScanner; import org.jabref.logic.directorylibrary.PdfEnrichmentTask; import org.jabref.logic.directorylibrary.PdfEntryFactory; +import org.jabref.logic.git.util.GitHandlerRegistry; import org.jabref.logic.l10n.Localization; import org.jabref.logic.util.BackgroundTask; import org.jabref.logic.util.TaskExecutor; @@ -41,7 +41,7 @@ public class OpenDirectoryLibraryAction extends SimpleCommand { private final StateManager stateManager; private final FileUpdateMonitor fileUpdateMonitor; private final BibEntryTypesManager entryTypesManager; - private final GuiUndoManager undoManager; + private final GitHandlerRegistry gitHandlerRegistry; private final ClipBoardManager clipBoardManager; private final TaskExecutor taskExecutor; @@ -52,7 +52,7 @@ public OpenDirectoryLibraryAction(LibraryTabContainer tabContainer, StateManager stateManager, FileUpdateMonitor fileUpdateMonitor, BibEntryTypesManager entryTypesManager, - GuiUndoManager undoManager, + GitHandlerRegistry gitHandlerRegistry, ClipBoardManager clipBoardManager, TaskExecutor taskExecutor) { this.tabContainer = tabContainer; @@ -62,7 +62,7 @@ public OpenDirectoryLibraryAction(LibraryTabContainer tabContainer, this.stateManager = stateManager; this.fileUpdateMonitor = fileUpdateMonitor; this.entryTypesManager = entryTypesManager; - this.undoManager = undoManager; + this.gitHandlerRegistry = gitHandlerRegistry; this.clipBoardManager = clipBoardManager; this.taskExecutor = taskExecutor; } @@ -117,9 +117,9 @@ private void showLibraryTab(DirectoryLibraryScanner.ScanResult scanResult, PdfEn stateManager, fileUpdateMonitor, entryTypesManager, - undoManager, clipBoardManager, - taskExecutor); + taskExecutor, + gitHandlerRegistry); tabContainer.addTab(libraryTab, true); if (!scanResult.pendingPdfImports().isEmpty()) { PdfEnrichmentTask enrichment = new PdfEnrichmentTask(scanResult.pendingPdfImports(), pdfEntryFactory, From c26e043f9b206d7b13a48e35cad6db7c7331494b Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Mon, 7 Sep 2026 11:13:38 +0200 Subject: [PATCH 13/16] Renumber the directory-as-library ADR to 0072 Upstream added a 0071 ADR after this stack claimed the number, so the MADR duplicate-ID check failed. Co-Authored-By: Claude Opus 4.8 --- ....md => 0072-directory-as-library-with-hayagriva-sidecars.md} | 2 +- docs/requirements/directory-library.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) rename docs/decisions/{0071-directory-as-library-with-hayagriva-sidecars.md => 0072-directory-as-library-with-hayagriva-sidecars.md} (99%) diff --git a/docs/decisions/0071-directory-as-library-with-hayagriva-sidecars.md b/docs/decisions/0072-directory-as-library-with-hayagriva-sidecars.md similarity index 99% rename from docs/decisions/0071-directory-as-library-with-hayagriva-sidecars.md rename to docs/decisions/0072-directory-as-library-with-hayagriva-sidecars.md index 5229eadcaf1f..79ff20e89107 100644 --- a/docs/decisions/0071-directory-as-library-with-hayagriva-sidecars.md +++ b/docs/decisions/0072-directory-as-library-with-hayagriva-sidecars.md @@ -1,5 +1,5 @@ --- -nav_order: 0071 +nav_order: 0072 parent: Decision Records --- diff --git a/docs/requirements/directory-library.md b/docs/requirements/directory-library.md index 7a54c2ae86d7..ef9577bd52e3 100644 --- a/docs/requirements/directory-library.md +++ b/docs/requirements/directory-library.md @@ -19,7 +19,7 @@ online and the metadata behind it fills only the fields the PDF did not provide. and `.md` files without a Hayagriva frontmatter are skipped; unparseable Hayagriva files are reported as warnings without aborting the scan. Scanning must not write or modify any file in the directory. -See [ADR 71](../decisions/0071-directory-as-library-with-hayagriva-sidecars.md) for more details. +See [ADR 72](../decisions/0072-directory-as-library-with-hayagriva-sidecars.md) for more details. Needs: impl From 9c5cde00c030f7388b8f141da19c8d7a4f5ce889 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Sun, 13 Sep 2026 03:24:15 +0200 Subject: [PATCH 14/16] Renumber directory-library ADR to 0075 Main took 0072. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CajtxjcCEjS87f4hoTBTJD --- ....md => 0075-directory-as-library-with-hayagriva-sidecars.md} | 2 +- docs/requirements/directory-library.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) rename docs/decisions/{0072-directory-as-library-with-hayagriva-sidecars.md => 0075-directory-as-library-with-hayagriva-sidecars.md} (99%) diff --git a/docs/decisions/0072-directory-as-library-with-hayagriva-sidecars.md b/docs/decisions/0075-directory-as-library-with-hayagriva-sidecars.md similarity index 99% rename from docs/decisions/0072-directory-as-library-with-hayagriva-sidecars.md rename to docs/decisions/0075-directory-as-library-with-hayagriva-sidecars.md index 79ff20e89107..4eab13d3ec34 100644 --- a/docs/decisions/0072-directory-as-library-with-hayagriva-sidecars.md +++ b/docs/decisions/0075-directory-as-library-with-hayagriva-sidecars.md @@ -1,5 +1,5 @@ --- -nav_order: 0072 +nav_order: 0075 parent: Decision Records --- diff --git a/docs/requirements/directory-library.md b/docs/requirements/directory-library.md index ef9577bd52e3..4ae2bf61319c 100644 --- a/docs/requirements/directory-library.md +++ b/docs/requirements/directory-library.md @@ -19,7 +19,7 @@ online and the metadata behind it fills only the fields the PDF did not provide. and `.md` files without a Hayagriva frontmatter are skipped; unparseable Hayagriva files are reported as warnings without aborting the scan. Scanning must not write or modify any file in the directory. -See [ADR 72](../decisions/0072-directory-as-library-with-hayagriva-sidecars.md) for more details. +See [ADR 75](../decisions/0075-directory-as-library-with-hayagriva-sidecars.md) for more details. Needs: impl From 81442e05528e178dfc4edd2df765448801f598d0 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Sun, 13 Sep 2026 21:55:08 +0200 Subject: [PATCH 15/16] Keep the library's own content out of the .gitignore scan filter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A folder whose `.gitignore` ignores everything — a `*` catch-all next to a `.gitkeep`, the common shape of a scratch PDF folder — opened as an empty library, because the scanner ran every PDF and sidecar through the gitignore filter. The filter now applies only to non-content files; hidden files and ignored subtrees are still skipped. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DDcHNMt9fPWnpYaHheFvry --- docs/requirements/directory-library.md | 10 +++++---- .../DirectoryLibraryScanner.java | 14 +++++++++++-- .../directorylibrary/PdfEnrichmentTask.java | 2 +- .../DirectoryLibraryScannerTest.java | 21 +++++++++++++++++-- 4 files changed, 38 insertions(+), 9 deletions(-) diff --git a/docs/requirements/directory-library.md b/docs/requirements/directory-library.md index 4ae2bf61319c..811a7cf9e03e 100644 --- a/docs/requirements/directory-library.md +++ b/docs/requirements/directory-library.md @@ -4,7 +4,7 @@ parent: Requirements # Directory library ## Directory scan builds the library from sidecars and PDFs -`req~directory-library.scan~5` +`req~directory-library.scan~6` Opening a directory as a library must fill the library from the directory tree: every Hayagriva `.yml`/`.yaml` file and every Markdown sidecar (`.md` opening with a Hayagriva YAML frontmatter @@ -15,9 +15,11 @@ linked to the sidecar's entry, and PDFs without a sidecar appear immediately as titled after the file; their metadata (embedded BibTeX, XMP, content heuristics — the standard PDF import pipeline) and a generated citation key arrive asynchronously after the library is shown, without replacing the entry instances. When the PDF yields no DOI, the DOI is looked up -online and the metadata behind it fills only the fields the PDF did not provide. Opening must not block on PDF parsing. Hidden files/directories, gitignored paths, `.yml` files not recognized as Hayagriva, -and `.md` files without a Hayagriva frontmatter -are skipped; unparseable Hayagriva files are reported as warnings without aborting the scan. +online and the metadata behind it fills only the fields the PDF did not provide. Opening must not block on PDF parsing. Hidden files/directories, gitignored non-content files and +gitignored subtrees, `.yml` files not recognized as Hayagriva, and `.md` files without a Hayagriva +frontmatter are skipped; the library's own sidecars and PDFs are never hidden by `.gitignore` (a +scratch PDF folder ignored by a catch-all `*` still opens). Unparseable Hayagriva files are +reported as warnings without aborting the scan. Scanning must not write or modify any file in the directory. See [ADR 75](../decisions/0075-directory-as-library-with-hayagriva-sidecars.md) for more details. diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java index b89659528e68..2a029717e33d 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryScanner.java @@ -39,7 +39,7 @@ /// [BibDatabaseContext] has [org.jabref.logic.shared.DatabaseLocation#DIRECTORY] and no /// database path; linked files are stored relative to the root, which is registered as the /// library-specific file directory. -// [impl->req~directory-library.scan~5] +// [impl->req~directory-library.scan~6] @NullMarked public class DirectoryLibraryScanner { @@ -140,10 +140,20 @@ public FileVisitResult preVisitDirectory(Path directory, BasicFileAttributes att @Override public FileVisitResult visitFile(Path file, BasicFileAttributes attributes) throws IOException { - if (isHidden(file) || !gitIgnoreFilter.accept(file)) { + if (isHidden(file)) { return FileVisitResult.CONTINUE; } String extension = FileUtil.getFileExtension(file).map(ext -> ext.toLowerCase(Locale.ROOT)).orElse(""); + boolean isLibraryContent = YAML_EXTENSIONS.contains(extension) + || MarkdownSidecar.MARKDOWN_EXTENSION.equals(extension) + || PDF_EXTENSION.equals(extension); + // The library's own content (sidecars and PDFs) is never hidden by .gitignore: a + // scratch PDF folder is commonly a `.gitignore` of `*` next to a `.gitkeep`, which + // would otherwise leave the whole library empty. .gitignore still governs every + // other file and, through preVisitDirectory, whole ignored subtrees. + if (!isLibraryContent && !gitIgnoreFilter.accept(file)) { + return FileVisitResult.CONTINUE; + } if (YAML_EXTENSIONS.contains(extension) || MarkdownSidecar.MARKDOWN_EXTENSION.equals(extension)) { sidecarFiles.add(file); } else if (PDF_EXTENSION.equals(extension)) { diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java index 89be73aaf67d..5ec89ef36b45 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/PdfEnrichmentTask.java @@ -18,7 +18,7 @@ /// /// Extraction runs on the background thread; entry mutations go through the injected /// marshaller (the JavaFX thread in the GUI). -// [impl->req~directory-library.scan~5] +// [impl->req~directory-library.scan~6] @NullMarked public class PdfEnrichmentTask extends BackgroundTask { diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java index b5eb2113ad7c..0aca0976bcc2 100644 --- a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibraryScannerTest.java @@ -275,14 +275,31 @@ void extensionsAreMatchedCaseInsensitively() throws IOException { } @Test - void gitignoredPdfIsNotPairedWithSidecar() throws IOException { + void gitignoredPdfIsStillPairedWithSidecar() throws IOException { + // PDFs are the library's own content, so .gitignore does not hide them Files.writeString(root.resolve(".gitignore"), "*.pdf\n"); Files.writeString(root.resolve("smith2020.yml"), ARTICLE_YAML); Files.createFile(root.resolve("smith2020.pdf")); ScanResult result = scan(); - assertEquals(List.of(), singleEntry(result).getFiles()); + assertEquals(List.of(new LinkedFile("", Path.of("smith2020.pdf"), "PDF")), singleEntry(result).getFiles()); + } + + @Test + void catchAllGitignoreDoesNotHideLibraryContent() throws IOException { + // A scratch PDF folder is commonly a `.gitignore` of `*` plus a `.gitkeep` + Files.writeString(root.resolve(".gitignore"), "*\n"); + Files.createFile(root.resolve(".gitkeep")); + Files.writeString(root.resolve("smith2020.yml"), ARTICLE_YAML); + Files.createFile(root.resolve("smith2020.pdf")); + Files.createFile(root.resolve("bare.pdf")); + + ScanResult result = scan(); + + List entries = result.databaseContext().getDatabase().getEntries(); + assertEquals(2, entries.size()); + assertEquals(List.of(new LinkedFile("", Path.of("smith2020.pdf"), "PDF")), entries.getFirst().getFiles()); } @Test From 64ce2ad8a5af33e17181aebf7cd0d1ea5c21a6b7 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Sun, 13 Sep 2026 23:10:28 +0200 Subject: [PATCH 16/16] Disable file-directory overrides for directory libraries A directory library resolves every linked file against its single root, so the library- and user-specific file directory overrides in the library properties do not apply. Both rows are now disabled when the library is a directory library. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DDcHNMt9fPWnpYaHheFvry --- .../general/GeneralPropertiesView.java | 11 +++++++++++ .../libraryproperties/general/GeneralProperties.fxml | 4 ++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/jabgui/src/main/java/org/jabref/gui/libraryproperties/general/GeneralPropertiesView.java b/jabgui/src/main/java/org/jabref/gui/libraryproperties/general/GeneralPropertiesView.java index d93aedbe247e..e745030452ff 100644 --- a/jabgui/src/main/java/org/jabref/gui/libraryproperties/general/GeneralPropertiesView.java +++ b/jabgui/src/main/java/org/jabref/gui/libraryproperties/general/GeneralPropertiesView.java @@ -19,6 +19,7 @@ import org.jabref.gui.util.ViewModelListCellFactory; import org.jabref.logic.l10n.Localization; import org.jabref.logic.preferences.CliPreferences; +import org.jabref.logic.shared.DatabaseLocation; import org.jabref.model.database.BibDatabaseContext; import org.jabref.model.database.BibDatabaseMode; @@ -39,6 +40,8 @@ public class GeneralPropertiesView extends AbstractPropertiesTabView { boolean isAbsolute = Path.of(newValue).isAbsolute(); libSpecificFileDirSwitchIcon.setGlyph(isAbsolute ? RELATIVE_PATH : ABSOLUTE_PATH); diff --git a/jabgui/src/main/resources/org/jabref/gui/libraryproperties/general/GeneralProperties.fxml b/jabgui/src/main/resources/org/jabref/gui/libraryproperties/general/GeneralProperties.fxml index b1d2e0bef31b..ab0665cee773 100644 --- a/jabgui/src/main/resources/org/jabref/gui/libraryproperties/general/GeneralProperties.fxml +++ b/jabgui/src/main/resources/org/jabref/gui/libraryproperties/general/GeneralProperties.fxml @@ -51,7 +51,7 @@ - -