From 7120843ca2319de4aaf45d77918c54fca5a88fa9 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Tue, 14 Jul 2026 00:07:23 +0200 Subject: [PATCH 1/9] Sync open directory libraries with external file changes (inbound) A DirectoryLibrarySynchronizer registers with the Commons-IO DirectoryMonitor (promoted from jabgui to jablib, per ADR-0030) and applies external create/change/delete events for .yml/.pdf files to the open library: changed sidecars update the existing BibEntry instances field by field, renames are detected via a grace window over the monitor's delete+create events and keep the affected entries, appearing PDFs link to their sidecar entry or become stubs, and disappearing PDFs unlink or remove their stub. All mutations carry EntriesEventSource.SHARED, and recordWrittenFile lets the upcoming write-back direction fingerprint its own writes so they are not re-imported. Late-joining observers are baselined with a silent first poll, otherwise the running monitor reports every existing file as created and duplicates the scanned entries. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 1 + docs/requirements/directory-library.md | 13 + .../main/java/org/jabref/gui/JabRefGUI.java | 2 +- .../main/java/org/jabref/gui/LibraryTab.java | 4 + .../jabref/gui/entryeditor/EntryEditor.java | 2 +- .../entryeditor/EntryEditorTabFactory.java | 2 +- .../gui/entryeditor/LatexCitationsTab.java | 2 +- .../LatexCitationsTabViewModel.java | 2 +- .../actions/OpenDirectoryLibraryAction.java | 15 + .../DirectoryLibraryCatalog.java | 23 + .../DirectoryLibrarySynchronizer.java | 498 ++++++++++++++++++ .../jabref/logic}/util/DirectoryMonitor.java | 2 +- .../model/database/BibDatabaseContext.java | 16 + .../DirectoryLibrarySynchronizerTest.java | 245 +++++++++ 14 files changed, 821 insertions(+), 6 deletions(-) create mode 100644 jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java rename {jabgui/src/main/java/org/jabref/gui => jablib/src/main/java/org/jabref/logic}/util/DirectoryMonitor.java (97%) create mode 100644 jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a1e46c9effa..5b8a3fdf6a98 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv ### Added +- Directory libraries now stay in sync with external file changes: creating, editing, deleting, or renaming `.yml`/`.pdf` files in the opened folder updates the open library live, and renames keep the affected entries (selection and undo history survive). [TODO] - We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear as stub entries titled after the file. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) - The HTTP import endpoint (`POST /libraries/{id}/entries`) now accepts CSL-JSON (`application/vnd.citationstyles.csl+json`), mapping each item to the correct entry type (e.g. conference paper, book chapter, thesis) via the citation-js-based mapping. [#16151](https://github.com/JabRef/jabref/pull/16151) - We added a new "Main" tab to the entry editor showing all fields of an entry in a single scrollable list, with one-click chips for adding optional fields and a free-form box for adding arbitrary fields. Identifiers, files and links, bibliometrics, comments, and meta fields (groups, owner, timestamps, special fields) live in collapsible sections — collapsed when empty — each offering chips for its unset fields. [#12711](https://github.com/JabRef/jabref/issues/12711) diff --git a/docs/requirements/directory-library.md b/docs/requirements/directory-library.md index 38e60f7897dd..8697164e0bcc 100644 --- a/docs/requirements/directory-library.md +++ b/docs/requirements/directory-library.md @@ -16,4 +16,17 @@ See [ADR 66](../decisions/0066-directory-as-library-with-hayagriva-sidecars.md) Needs: impl +## External file changes appear live in an open directory library +`req~directory-library.inbound-sync~1` + +While a directory library is open, external creation, modification, deletion, and renaming of +`.yml`/`.yaml`/`.pdf` files under its root must be reflected in the open library. Changed +entries are updated in place (the entry identity is preserved), renames are detected via a +grace window over the monitor's delete + create events and keep the affected entries, and files +written by JabRef itself are recognized by fingerprint and not re-imported. All resulting +database mutations carry a non-local event source so the future write-back direction can ignore +them. + +Needs: impl + diff --git a/jabgui/src/main/java/org/jabref/gui/JabRefGUI.java b/jabgui/src/main/java/org/jabref/gui/JabRefGUI.java index 7330c32a7011..22ff7fb80fcd 100644 --- a/jabgui/src/main/java/org/jabref/gui/JabRefGUI.java +++ b/jabgui/src/main/java/org/jabref/gui/JabRefGUI.java @@ -33,7 +33,6 @@ import org.jabref.gui.theme.ThemeManager; import org.jabref.gui.undo.CountingUndoManager; import org.jabref.gui.util.DefaultFileUpdateMonitor; -import org.jabref.gui.util.DirectoryMonitor; import org.jabref.gui.util.UiTaskExecutor; import org.jabref.http.manager.HttpServerManager; import org.jabref.languageserver.controller.LanguageServerController; @@ -52,6 +51,7 @@ import org.jabref.logic.search.sqlbased.IndexManager; import org.jabref.logic.search.sqlbased.PostgresServer; import org.jabref.logic.util.BuildInfo; +import org.jabref.logic.util.DirectoryMonitor; import org.jabref.logic.util.FallbackExceptionHandler; import org.jabref.logic.util.HeadlessExecutorService; import org.jabref.logic.util.TaskExecutor; diff --git a/jabgui/src/main/java/org/jabref/gui/LibraryTab.java b/jabgui/src/main/java/org/jabref/gui/LibraryTab.java index 84a8ad03cd17..66ee3c6ffb6d 100644 --- a/jabgui/src/main/java/org/jabref/gui/LibraryTab.java +++ b/jabgui/src/main/java/org/jabref/gui/LibraryTab.java @@ -739,6 +739,10 @@ private void onClosed(Event event) { if (dataLoadingTask != null) { dataLoadingTask.cancel(); } + if (bibDatabaseContext.getLocation() == DatabaseLocation.DIRECTORY) { + // Stops the directory watcher and shuts the synchronizer down + bibDatabaseContext.convertToLocalDatabase(); + } if (bibDatabaseContext.getLocation() == DatabaseLocation.SHARED) { bibDatabaseContext.convertToLocalDatabase(); bibDatabaseContext.getDBMSSynchronizer().closeSharedDatabase(); diff --git a/jabgui/src/main/java/org/jabref/gui/entryeditor/EntryEditor.java b/jabgui/src/main/java/org/jabref/gui/entryeditor/EntryEditor.java index eba6fc8da1fb..e6225a2a1d50 100644 --- a/jabgui/src/main/java/org/jabref/gui/entryeditor/EntryEditor.java +++ b/jabgui/src/main/java/org/jabref/gui/entryeditor/EntryEditor.java @@ -37,7 +37,6 @@ import org.jabref.gui.undo.RedoAction; import org.jabref.gui.undo.UndoAction; import org.jabref.gui.util.BaseDialog; -import org.jabref.gui.util.DirectoryMonitor; import org.jabref.gui.util.DragDrop; import org.jabref.logic.ai.AiService; import org.jabref.logic.citation.SearchCitationsRelationsService; @@ -45,6 +44,7 @@ import org.jabref.logic.importer.EntryBasedFetcher; import org.jabref.logic.journals.JournalAbbreviationRepository; import org.jabref.logic.util.BuildInfo; +import org.jabref.logic.util.DirectoryMonitor; import org.jabref.logic.util.TaskExecutor; import org.jabref.model.entry.BibEntry; import org.jabref.model.entry.BibEntryTypesManager; diff --git a/jabgui/src/main/java/org/jabref/gui/entryeditor/EntryEditorTabFactory.java b/jabgui/src/main/java/org/jabref/gui/entryeditor/EntryEditorTabFactory.java index b86a6779304a..8970ee27fed4 100644 --- a/jabgui/src/main/java/org/jabref/gui/entryeditor/EntryEditorTabFactory.java +++ b/jabgui/src/main/java/org/jabref/gui/entryeditor/EntryEditorTabFactory.java @@ -14,10 +14,10 @@ import org.jabref.gui.undo.CountingUndoManager; import org.jabref.gui.undo.RedoAction; import org.jabref.gui.undo.UndoAction; -import org.jabref.gui.util.DirectoryMonitor; import org.jabref.logic.citation.SearchCitationsRelationsService; import org.jabref.logic.journals.JournalAbbreviationRepository; import org.jabref.logic.util.BuildInfo; +import org.jabref.logic.util.DirectoryMonitor; import org.jabref.logic.util.TaskExecutor; import org.jabref.model.entry.BibEntryTypesManager; import org.jabref.model.util.FileUpdateMonitor; diff --git a/jabgui/src/main/java/org/jabref/gui/entryeditor/LatexCitationsTab.java b/jabgui/src/main/java/org/jabref/gui/entryeditor/LatexCitationsTab.java index 39510d0743eb..4bbd4c48ca6f 100644 --- a/jabgui/src/main/java/org/jabref/gui/entryeditor/LatexCitationsTab.java +++ b/jabgui/src/main/java/org/jabref/gui/entryeditor/LatexCitationsTab.java @@ -20,8 +20,8 @@ import org.jabref.gui.icon.IconTheme; import org.jabref.gui.preferences.GuiPreferences; import org.jabref.gui.texparser.CitationsDisplay; -import org.jabref.gui.util.DirectoryMonitor; import org.jabref.logic.l10n.Localization; +import org.jabref.logic.util.DirectoryMonitor; import org.jabref.model.entry.BibEntry; import com.tobiasdiez.easybind.EasyBind; diff --git a/jabgui/src/main/java/org/jabref/gui/entryeditor/LatexCitationsTabViewModel.java b/jabgui/src/main/java/org/jabref/gui/entryeditor/LatexCitationsTabViewModel.java index dd925cbc6f64..96eaa6fa1334 100644 --- a/jabgui/src/main/java/org/jabref/gui/entryeditor/LatexCitationsTabViewModel.java +++ b/jabgui/src/main/java/org/jabref/gui/entryeditor/LatexCitationsTabViewModel.java @@ -27,10 +27,10 @@ import org.jabref.gui.push.GuiPushToTeXstudio; import org.jabref.gui.texparser.CitationsDisplay; import org.jabref.gui.util.DirectoryDialogConfiguration; -import org.jabref.gui.util.DirectoryMonitor; import org.jabref.gui.util.UiTaskExecutor; import org.jabref.logic.l10n.Localization; import org.jabref.logic.texparser.DefaultLatexParser; +import org.jabref.logic.util.DirectoryMonitor; import org.jabref.logic.util.io.FileUtil; import org.jabref.model.database.BibDatabaseContext; import org.jabref.model.entry.BibEntry; diff --git a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java index 5bb036cfbc79..6048f79e88c3 100644 --- a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java +++ b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java @@ -12,14 +12,20 @@ import org.jabref.gui.clipboard.ClipBoardManager; import org.jabref.gui.preferences.GuiPreferences; import org.jabref.gui.util.DirectoryDialogConfiguration; +import org.jabref.gui.util.UiTaskExecutor; import org.jabref.logic.ai.AiService; import org.jabref.logic.directorylibrary.DirectoryLibraryScanner; +import org.jabref.logic.directorylibrary.DirectoryLibrarySynchronizer; import org.jabref.logic.l10n.Localization; import org.jabref.logic.util.BackgroundTask; +import org.jabref.logic.util.DirectoryMonitor; import org.jabref.logic.util.TaskExecutor; +import org.jabref.model.database.BibDatabaseContext; import org.jabref.model.entry.BibEntryTypesManager; import org.jabref.model.util.FileUpdateMonitor; +import com.airhacks.afterburner.injection.Injector; + /// Opens a directory as a library: the main table fills from the Hayagriva `.yml` sidecars and /// `.pdf` files found in the directory tree (see [DirectoryLibraryScanner]). public class OpenDirectoryLibraryAction extends SimpleCommand { @@ -93,6 +99,15 @@ private void showLibraryTab(DirectoryLibraryScanner.ScanResult scanResult) { clipBoardManager, taskExecutor); tabContainer.addTab(libraryTab, true); + // No change event follows the synchronous tab creation, so set the initial title here + libraryTab.updateTabTitle(false); + + BibDatabaseContext databaseContext = scanResult.databaseContext(); + DirectoryLibrarySynchronizer synchronizer = new DirectoryLibrarySynchronizer( + databaseContext, scanResult.catalog(), UiTaskExecutor::runInJavaFXThread); + databaseContext.attachDirectorySynchronizer(synchronizer); + synchronizer.startWatching(Injector.instantiateModelOrService(DirectoryMonitor.class)); + if (!scanResult.warnings().isEmpty()) { dialogService.showWarningDialogAndWait( Localization.lang("Open folder as library"), diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryCatalog.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryCatalog.java index 76b38d469522..fcb1f0dfc104 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryCatalog.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryCatalog.java @@ -40,4 +40,27 @@ public Optional sourceOf(BibEntry entry) { public List entryIdsIn(Path yamlFile) { return List.copyOf(entryIdsByFile.getOrDefault(yamlFile, List.of())); } + + /// Re-homes all entries of `oldFile` to `newFile` (a rename/move on disk). + public void relocateFile(Path oldFile, Path newFile) { + List entryIds = entryIdsByFile.remove(oldFile); + if (entryIds == null) { + return; + } + entryIdsByFile.put(newFile, entryIds); + entryIds.forEach(entryId -> { + EntrySource source = sourceByEntryId.get(entryId); + if (source != null) { + sourceByEntryId.put(entryId, new EntrySource(newFile, source.hayagrivaKey())); + } + }); + } + + /// Forgets all entries of the given file (deleted on disk or re-registered afterwards). + public void removeFile(Path yamlFile) { + List entryIds = entryIdsByFile.remove(yamlFile); + if (entryIds != null) { + entryIds.forEach(sourceByEntryId::remove); + } + } } diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java new file mode 100644 index 000000000000..6b581d09b0df --- /dev/null +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java @@ -0,0 +1,498 @@ +package org.jabref.logic.directorylibrary; + +import java.io.BufferedReader; +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HexFormat; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Optional; +import java.util.SequencedMap; +import java.util.Set; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; +import java.util.function.Consumer; + +import org.jabref.logic.bibtex.FileFieldWriter; +import org.jabref.logic.importer.ParserResult; +import org.jabref.logic.importer.fileformat.HayagrivaImporter; +import org.jabref.logic.util.DirectoryMonitor; +import org.jabref.logic.util.StandardFileType; +import org.jabref.logic.util.io.FileUtil; +import org.jabref.model.database.BibDatabaseContext; +import org.jabref.model.entry.BibEntry; +import org.jabref.model.entry.LinkedFile; +import org.jabref.model.entry.event.EntriesEventSource; +import org.jabref.model.entry.field.StandardField; +import org.jabref.model.entry.types.StandardEntryType; + +import org.apache.commons.io.IOCase; +import org.apache.commons.io.filefilter.FileFilterUtils; +import org.apache.commons.io.filefilter.IOFileFilter; +import org.apache.commons.io.monitor.FileAlterationListener; +import org.apache.commons.io.monitor.FileAlterationObserver; +import org.apache.commons.io.monitor.FileEntry; +import org.jspecify.annotations.NullMarked; +import org.jspecify.annotations.Nullable; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/// Keeps an open directory library in sync with external file changes (inbound direction: +/// file system to [BibDatabaseContext]). Registered as a [FileAlterationListener] with the +/// polling [DirectoryMonitor]; all event handling is serialized on a single "directory-sync" +/// executor, and model mutations are marshalled through the injected `modelUpdateMarshaller` +/// (the GUI passes the JavaFX thread executor). +/// +/// All database mutations use [EntriesEventSource#SHARED] so that the future write-back +/// direction can ignore them (same echo-prevention policy as the shared-SQL synchronizer). +/// Conversely, [#recordWrittenFile] lets the write-back direction register a fingerprint of +/// its own writes, which this class then swallows instead of re-importing. +/// +/// The file monitor reports renames as delete + create. Deletions are therefore staged for a +/// grace period spanning two poll cycles: a create whose parsed entries equal a staged +/// deletion's entries is treated as a move (the [BibEntry] instances survive, preserving +/// selection and undo history); only unmatched deletions are committed. +// [impl->req~directory-library.inbound-sync~1] +@NullMarked +public class DirectoryLibrarySynchronizer implements FileAlterationListener { + + private static final Logger LOGGER = LoggerFactory.getLogger(DirectoryLibrarySynchronizer.class); + + /// Two poll cycles of [DirectoryMonitor] (1 s each), so a rename's create event can arrive + /// in the poll cycle after its delete event. + private static final Duration RENAME_GRACE = Duration.ofMillis(2500); + + private static final Set YAML_EXTENSIONS = Set.of("yml", "yaml"); + private static final String PDF_EXTENSION = "pdf"; + + private final BibDatabaseContext databaseContext; + private final DirectoryLibraryCatalog catalog; + private final Path root; + private final Consumer modelUpdateMarshaller; + private final Clock clock; + private final HayagrivaImporter importer = new HayagrivaImporter(); + private final ScheduledExecutorService syncExecutor; + + private final Map stagedDeletions = new HashMap<>(); + private final Map lastWrittenFingerprints = new HashMap<>(); + + private @Nullable FileAlterationObserver observer; + private @Nullable DirectoryMonitor directoryMonitor; + + private record StagedDeletion(List entries, Instant expiry) { + } + + public DirectoryLibrarySynchronizer(BibDatabaseContext databaseContext, + DirectoryLibraryCatalog catalog, + Consumer modelUpdateMarshaller) { + this(databaseContext, catalog, modelUpdateMarshaller, Clock.systemUTC()); + } + + DirectoryLibrarySynchronizer(BibDatabaseContext databaseContext, + DirectoryLibraryCatalog catalog, + Consumer modelUpdateMarshaller, + Clock clock) { + this.databaseContext = databaseContext; + this.catalog = catalog; + this.root = databaseContext.getDirectoryLibraryRoot().orElseThrow( + () -> new IllegalArgumentException("Context is not a directory library")); + this.modelUpdateMarshaller = modelUpdateMarshaller; + this.clock = clock; + this.syncExecutor = Executors.newSingleThreadScheduledExecutor(runnable -> { + Thread thread = new Thread(runnable, "directory-sync"); + thread.setDaemon(true); + return thread; + }); + } + + public void startWatching(DirectoryMonitor monitor) { + this.directoryMonitor = monitor; + IOFileFilter relevantFiles = FileFilterUtils.or( + FileFilterUtils.directoryFileFilter(), + FileFilterUtils.suffixFileFilter(".yml", IOCase.INSENSITIVE), + FileFilterUtils.suffixFileFilter(".yaml", IOCase.INSENSITIVE), + FileFilterUtils.suffixFileFilter(".pdf", IOCase.INSENSITIVE)); + IOFileFilter notHidden = FileFilterUtils.notFileFilter(FileFilterUtils.prefixFileFilter(".")); + observer = FileAlterationObserver.builder() + .setRootEntry(new FileEntry(root.toFile())) + .setFileFilter(FileFilterUtils.and(notHidden, relevantFiles)) + .getUnchecked(); + // The monitor is already running and never initializes late-joining observers, so the + // first poll would report every existing file as created. Checking once without any + // listener attached takes the baseline snapshot silently. + observer.checkAndNotify(); + monitor.addObserver(observer, this); + } + + public void shutdown() { + if (observer != null && directoryMonitor != null) { + directoryMonitor.removeObserver(observer); + } + syncExecutor.shutdown(); + } + + /// Registers the fingerprint of a file this application just wrote itself, so the next + /// change event for it is recognized as a self-echo and not re-imported. Consumed on match. + public synchronized void recordWrittenFile(Path file, byte[] content) { + lastWrittenFingerprints.put(file.toAbsolutePath().normalize(), hash(content)); + } + + @Override + public void onFileCreate(File file) { + syncExecutor.execute(() -> handleFileCreated(file.toPath())); + } + + @Override + public void onFileChange(File file) { + syncExecutor.execute(() -> handleFileChanged(file.toPath())); + } + + @Override + public void onFileDelete(File file) { + syncExecutor.execute(() -> handleFileDeleted(file.toPath())); + } + + @Override + public void onDirectoryCreate(File directory) { + // files inside are reported individually + } + + @Override + public void onDirectoryChange(File directory) { + // files inside are reported individually + } + + @Override + public void onDirectoryDelete(File directory) { + // files inside are reported individually + } + + @Override + public void onStart(FileAlterationObserver observer) { + // no bookkeeping per scan round needed + } + + @Override + public void onStop(FileAlterationObserver observer) { + syncExecutor.execute(this::commitExpiredStagedDeletions); + } + + synchronized void handleFileCreated(Path file) { + commitExpiredStagedDeletions(); + if (isYaml(file)) { + if (consumeSelfEcho(file)) { + return; + } + importFile(file); + } else if (isPdf(file)) { + handlePdfCreated(file); + } + } + + synchronized void handleFileChanged(Path file) { + commitExpiredStagedDeletions(); + if (!isYaml(file) || consumeSelfEcho(file)) { + return; + } + List knownEntries = entriesOf(file); + if (knownEntries.isEmpty()) { + importFile(file); + return; + } + if (!looksLikeHayagriva(file)) { + // The file stopped being a Hayagriva file (e.g. replaced by unrelated YAML) + removeEntries(knownEntries, file); + return; + } + Optional> parsed = parse(file); + if (parsed.isEmpty()) { + LOGGER.warn("Not applying changes of unparseable Hayagriva file {}", file); + return; + } + applyChangedFile(file, knownEntries, parsed.get()); + } + + synchronized void handleFileDeleted(Path file) { + commitExpiredStagedDeletions(); + if (isYaml(file)) { + List entries = entriesOf(file); + if (entries.isEmpty()) { + return; + } + stagedDeletions.put(file, new StagedDeletion(entries, clock.instant().plus(RENAME_GRACE))); + syncExecutor.schedule(this::commitExpiredStagedDeletions, + RENAME_GRACE.toMillis() + 100, TimeUnit.MILLISECONDS); + } else if (isPdf(file)) { + handlePdfDeleted(file); + } + } + + synchronized void commitExpiredStagedDeletions() { + Instant now = clock.instant(); + List> expired = stagedDeletions.entrySet().stream() + .filter(staged -> !staged.getValue().expiry().isAfter(now)) + .toList(); + for (Map.Entry staged : expired) { + stagedDeletions.remove(staged.getKey()); + removeEntries(staged.getValue().entries(), staged.getKey()); + } + } + + private void importFile(Path file) { + if (!entriesOf(file).isEmpty()) { + // Already known (e.g. a create event for a file the scan covered) — diff instead + handleFileChanged(file); + return; + } + if (!looksLikeHayagriva(file)) { + return; + } + Optional> parsed = parse(file); + if (parsed.isEmpty() || parsed.get().isEmpty()) { + return; + } + List newEntries = parsed.get(); + + // A staged deletion with equal content is this file being moved, not new content + Optional movedFrom = stagedDeletions.entrySet().stream() + .filter(staged -> entriesMatch(staged.getValue().entries(), newEntries)) + .map(Map.Entry::getKey) + .findFirst(); + if (movedFrom.isPresent()) { + stagedDeletions.remove(movedFrom.get()); + catalog.relocateFile(movedFrom.get(), file); + LOGGER.debug("Detected move {} -> {}", movedFrom.get(), file); + return; + } + + newEntries.forEach(entry -> catalog.register(entry, file, entry.getCitationKey().orElse(""))); + // Safe without event source: the entry is not yet inserted, so no listeners see this + findPairedPdf(file).ifPresent(pdf -> newEntries.getFirst() + .addFile(new LinkedFile("", root.relativize(pdf), StandardFileType.PDF.getName()))); + modelUpdateMarshaller.accept(() -> + databaseContext.getDatabase().insertEntries(newEntries, EntriesEventSource.SHARED)); + } + + private void applyChangedFile(Path file, List knownEntries, List parsedEntries) { + SequencedMap knownByKey = byCitationKey(knownEntries); + SequencedMap parsedByKey = byCitationKey(parsedEntries); + + List toInsert = new ArrayList<>(); + List toRemove = new ArrayList<>(); + List fieldUpdates = new ArrayList<>(); + + parsedByKey.forEach((key, parsedEntry) -> { + BibEntry knownEntry = knownByKey.get(key); + if (knownEntry == null) { + catalog.register(parsedEntry, file, key); + toInsert.add(parsedEntry); + } else { + fieldUpdates.add(() -> copyContent(parsedEntry, knownEntry)); + } + }); + knownByKey.forEach((key, knownEntry) -> { + if (!parsedByKey.containsKey(key)) { + toRemove.add(knownEntry); + } + }); + + modelUpdateMarshaller.accept(() -> { + fieldUpdates.forEach(Runnable::run); + if (!toInsert.isEmpty()) { + databaseContext.getDatabase().insertEntries(toInsert, EntriesEventSource.SHARED); + } + if (!toRemove.isEmpty()) { + databaseContext.getDatabase().removeEntries(toRemove, EntriesEventSource.SHARED); + } + }); + catalog.removeFile(file); + parsedByKey.forEach((key, parsedEntry) -> { + BibEntry target = knownByKey.getOrDefault(key, parsedEntry); + catalog.register(target, file, key); + }); + } + + /// Applies `source`'s type and fields onto `target` without replacing the instance, so + /// selection, undo history, and group membership survive external edits. + private void copyContent(BibEntry source, BibEntry target) { + if (!target.getType().equals(source.getType())) { + target.setType(source.getType(), EntriesEventSource.SHARED); + } + // The PDF link is maintained by this synchronizer, not by the file content + Optional preservedFiles = target.getField(StandardField.FILE); + source.getFields().forEach(field -> + source.getField(field).ifPresent(value -> target.setField(field, value, EntriesEventSource.SHARED))); + target.getFields().stream() + .filter(field -> StandardField.FILE != field) + .filter(field -> source.getField(field).isEmpty()) + .toList() + .forEach(field -> target.clearField(field, EntriesEventSource.SHARED)); + preservedFiles.ifPresent(files -> target.setField(StandardField.FILE, files, EntriesEventSource.SHARED)); + } + + private void handlePdfCreated(Path pdf) { + Optional sidecarEntry = findSidecarEntry(pdf); + if (sidecarEntry.isPresent()) { + BibEntry entry = sidecarEntry.get(); + if (entry.getFiles().isEmpty()) { + List files = List.of(new LinkedFile("", root.relativize(pdf), StandardFileType.PDF.getName())); + modelUpdateMarshaller.accept(() -> + entry.setField(StandardField.FILE, FileFieldWriter.getStringRepresentation(files), EntriesEventSource.SHARED)); + } + return; + } + BibEntry stub = new BibEntry(StandardEntryType.Misc) + .withField(StandardField.TITLE, FileUtil.getBaseName(pdf)); + stub.addFile(new LinkedFile("", root.relativize(pdf), StandardFileType.PDF.getName())); + modelUpdateMarshaller.accept(() -> + databaseContext.getDatabase().insertEntries(List.of(stub), EntriesEventSource.SHARED)); + } + + private void handlePdfDeleted(Path pdf) { + String relativeLink = root.relativize(pdf).toString(); + List linking = databaseContext.getDatabase().getEntries().stream() + .filter(entry -> entry.getFiles().stream() + .anyMatch(linked -> relativeLink.equals(linked.getLink()))) + .toList(); + for (BibEntry entry : linking) { + boolean isStub = catalog.sourceOf(entry).isEmpty(); + modelUpdateMarshaller.accept(() -> { + if (isStub) { + databaseContext.getDatabase().removeEntries(List.of(entry), EntriesEventSource.SHARED); + } else { + List remaining = entry.getFiles().stream() + .filter(linked -> !relativeLink.equals(linked.getLink())) + .toList(); + entry.setField(StandardField.FILE, FileFieldWriter.getStringRepresentation(remaining), EntriesEventSource.SHARED); + } + }); + } + } + + private void removeEntries(List entries, Path file) { + catalog.removeFile(file); + modelUpdateMarshaller.accept(() -> + databaseContext.getDatabase().removeEntries(entries, EntriesEventSource.SHARED)); + } + + private List entriesOf(Path file) { + List ids = catalog.entryIdsIn(file); + if (ids.isEmpty()) { + return List.of(); + } + Map byId = new HashMap<>(); + databaseContext.getDatabase().getEntries().forEach(entry -> byId.put(entry.getId(), entry)); + return ids.stream().flatMap(id -> Optional.ofNullable(byId.get(id)).stream()).toList(); + } + + private static SequencedMap byCitationKey(List entries) { + SequencedMap byKey = new LinkedHashMap<>(); + entries.forEach(entry -> byKey.putIfAbsent(entry.getCitationKey().orElse(""), entry)); + return byKey; + } + + private static boolean entriesMatch(List staged, List parsed) { + if (staged.size() != parsed.size()) { + return false; + } + for (int i = 0; i < staged.size(); i++) { + if (!staged.get(i).equals(parsed.get(i))) { + return false; + } + } + return true; + } + + private Optional> parse(Path file) { + try { + ParserResult parserResult = importer.importDatabase(file); + if (parserResult.isInvalid()) { + return Optional.empty(); + } + return Optional.of(parserResult.getDatabase().getEntries()); + } catch (IOException e) { + LOGGER.warn("Could not read {}", file, e); + return Optional.empty(); + } + } + + private boolean looksLikeHayagriva(Path file) { + try (BufferedReader reader = Files.newBufferedReader(file, StandardCharsets.UTF_8)) { + return importer.isRecognizedFormat(reader); + } catch (IOException e) { + LOGGER.warn("Could not read {}", file, e); + return false; + } + } + + private Optional findSidecarEntry(Path pdf) { + Path parent = pdf.getParent(); + if (parent == null) { + return Optional.empty(); + } + String baseName = FileUtil.getBaseName(pdf); + for (String extension : YAML_EXTENSIONS) { + List entries = entriesOf(parent.resolve(baseName + "." + extension)); + if (!entries.isEmpty()) { + return Optional.of(entries.getFirst()); + } + } + return Optional.empty(); + } + + private Optional findPairedPdf(Path yamlFile) { + Path parent = yamlFile.getParent(); + if (parent == null) { + return Optional.empty(); + } + Path pdf = parent.resolve(FileUtil.getBaseName(yamlFile) + ".pdf"); + return Files.exists(pdf) ? Optional.of(pdf) : Optional.empty(); + } + + private boolean consumeSelfEcho(Path file) { + Path normalized = file.toAbsolutePath().normalize(); + String recorded = lastWrittenFingerprints.get(normalized); + if (recorded == null) { + return false; + } + try { + String current = hash(Files.readAllBytes(file)); + if (recorded.equals(current)) { + lastWrittenFingerprints.remove(normalized); + return true; + } + } catch (IOException e) { + LOGGER.debug("Could not fingerprint {}", file, e); + } + return false; + } + + private static String hash(byte[] content) { + try { + return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(content)); + } catch (NoSuchAlgorithmException e) { + throw new AssertionError("SHA-256 is guaranteed to be available", e); + } + } + + private static boolean isYaml(Path file) { + return YAML_EXTENSIONS.contains(FileUtil.getFileExtension(file).orElse("").toLowerCase(Locale.ROOT)); + } + + private static boolean isPdf(Path file) { + return PDF_EXTENSION.equals(FileUtil.getFileExtension(file).orElse("").toLowerCase(Locale.ROOT)); + } +} diff --git a/jabgui/src/main/java/org/jabref/gui/util/DirectoryMonitor.java b/jablib/src/main/java/org/jabref/logic/util/DirectoryMonitor.java similarity index 97% rename from jabgui/src/main/java/org/jabref/gui/util/DirectoryMonitor.java rename to jablib/src/main/java/org/jabref/logic/util/DirectoryMonitor.java index def65f9549f2..75a0d66dec6a 100644 --- a/jabgui/src/main/java/org/jabref/gui/util/DirectoryMonitor.java +++ b/jablib/src/main/java/org/jabref/logic/util/DirectoryMonitor.java @@ -1,4 +1,4 @@ -package org.jabref.gui.util; +package org.jabref.logic.util; import org.apache.commons.io.monitor.FileAlterationListener; import org.apache.commons.io.monitor.FileAlterationMonitor; diff --git a/jablib/src/main/java/org/jabref/model/database/BibDatabaseContext.java b/jablib/src/main/java/org/jabref/model/database/BibDatabaseContext.java index 648050f132ea..a64a61e06acb 100644 --- a/jablib/src/main/java/org/jabref/model/database/BibDatabaseContext.java +++ b/jablib/src/main/java/org/jabref/model/database/BibDatabaseContext.java @@ -20,6 +20,7 @@ import org.jabref.logic.JabRefException; import org.jabref.logic.crawler.Crawler; import org.jabref.logic.crawler.StudyRepository; +import org.jabref.logic.directorylibrary.DirectoryLibrarySynchronizer; import org.jabref.logic.importer.ImportFormatPreferences; import org.jabref.logic.importer.ParserResult; import org.jabref.logic.importer.fileformat.BibtexParser; @@ -73,6 +74,9 @@ public class BibDatabaseContext { @Nullable private Path directoryLibraryRoot; + @Nullable + private DirectoryLibrarySynchronizer directorySynchronizer; + private DatabaseLocation location; public BibDatabaseContext() { @@ -283,6 +287,14 @@ public Optional getDirectoryLibraryRoot() { return Optional.ofNullable(directoryLibraryRoot); } + public void attachDirectorySynchronizer(DirectoryLibrarySynchronizer directorySynchronizer) { + this.directorySynchronizer = directorySynchronizer; + } + + public @Nullable DirectoryLibrarySynchronizer getDirectorySynchronizer() { + return directorySynchronizer; + } + public void convertToLocalDatabase() { if (dbmsListener != null && (location == DatabaseLocation.SHARED)) { if (dbmsSynchronizer != null) { @@ -290,6 +302,10 @@ public void convertToLocalDatabase() { } dbmsListener.shutdown(); } + if (directorySynchronizer != null) { + directorySynchronizer.shutdown(); + directorySynchronizer = null; + } this.directoryLibraryRoot = null; this.location = DatabaseLocation.LOCAL; diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java new file mode 100644 index 000000000000..a0b02e9afffd --- /dev/null +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java @@ -0,0 +1,245 @@ +package org.jabref.logic.directorylibrary; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.util.List; +import java.util.Optional; + +import org.jabref.model.database.BibDatabaseContext; +import org.jabref.model.entry.BibEntry; +import org.jabref.model.entry.field.StandardField; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class DirectoryLibrarySynchronizerTest { + + private static final String ARTICLE_YAML = """ + smith2020: + type: article + title: A Test Article + author: Smith, Jane + note: first version + """; + + /// Deterministic clock for the rename grace window. + private static final class SteppingClock extends Clock { + private Instant now = Instant.parse("2026-07-13T12:00:00Z"); + + private void advance(Duration duration) { + now = now.plus(duration); + } + + @Override + public Instant instant() { + return now; + } + + @Override + public ZoneId getZone() { + return ZoneOffset.UTC; + } + + @Override + public Clock withZone(ZoneId zone) { + return this; + } + } + + @TempDir + Path root; + + private final SteppingClock clock = new SteppingClock(); + + private BibDatabaseContext context; + private DirectoryLibrarySynchronizer synchronizer; + + private void openLibrary() throws IOException { + DirectoryLibraryScanner.ScanResult scanResult = new DirectoryLibraryScanner().scan(root); + context = scanResult.databaseContext(); + synchronizer = new DirectoryLibrarySynchronizer(context, scanResult.catalog(), Runnable::run, clock); + } + + private List entries() { + return context.getDatabase().getEntries(); + } + + @Test + void externallyCreatedSidecarAddsEntryAndLinksPdf() throws IOException { + Files.createFile(root.resolve("smith2020.pdf")); + openLibrary(); + // The bare PDF became a stub during the scan; an appearing sidecar adds its entry + Path sidecar = root.resolve("smith2020.yml"); + Files.writeString(sidecar, ARTICLE_YAML); + + synchronizer.handleFileCreated(sidecar); + + assertEquals(2, entries().size()); + BibEntry added = entries().getLast(); + assertEquals(Optional.of("smith2020"), added.getCitationKey()); + assertEquals(1, added.getFiles().size()); + } + + @Test + void externalChangeUpdatesTheSameEntryInstance() throws IOException { + Path sidecar = root.resolve("smith2020.yml"); + Files.writeString(sidecar, ARTICLE_YAML); + Files.createFile(root.resolve("smith2020.pdf")); + openLibrary(); + BibEntry entry = entries().getFirst(); + + Files.writeString(sidecar, ARTICLE_YAML.replace("first version", "second version")); + synchronizer.handleFileChanged(sidecar); + + assertEquals(1, entries().size()); + assertSame(entry, entries().getFirst()); + assertEquals(Optional.of("second version"), entry.getField(StandardField.NOTE)); + assertEquals(1, entry.getFiles().size()); + } + + @Test + void externalChangeAddsAndRemovesEntriesOfMultiEntryFile() throws IOException { + Path file = root.resolve("collection.yml"); + Files.writeString(file, """ + first: + type: article + title: First + second: + type: article + title: Second + """); + openLibrary(); + + Files.writeString(file, """ + first: + type: article + title: First + third: + type: article + title: Third + """); + synchronizer.handleFileChanged(file); + + assertEquals(List.of(Optional.of("first"), Optional.of("third")), + entries().stream().map(BibEntry::getCitationKey).toList()); + } + + @Test + void externalDeleteRemovesEntriesOnlyAfterGraceWindow() throws IOException { + Path sidecar = root.resolve("smith2020.yml"); + Files.writeString(sidecar, ARTICLE_YAML); + openLibrary(); + + Files.delete(sidecar); + synchronizer.handleFileDeleted(sidecar); + assertEquals(1, entries().size()); + + clock.advance(Duration.ofSeconds(3)); + synchronizer.commitExpiredStagedDeletions(); + assertEquals(0, entries().size()); + } + + @Test + void renameIsDetectedAsMoveAndPreservesEntryInstance() throws IOException { + Path oldFile = root.resolve("smith2020.yml"); + Files.writeString(oldFile, ARTICLE_YAML); + openLibrary(); + BibEntry entry = entries().getFirst(); + + Path newFile = root.resolve("renamed.yml"); + Files.move(oldFile, newFile); + synchronizer.handleFileDeleted(oldFile); + synchronizer.handleFileCreated(newFile); + + clock.advance(Duration.ofSeconds(3)); + synchronizer.commitExpiredStagedDeletions(); + + assertEquals(1, entries().size()); + assertSame(entry, entries().getFirst()); + } + + @Test + void selfWrittenFileIsNotReimported() throws IOException { + openLibrary(); + Path sidecar = root.resolve("smith2020.yml"); + byte[] content = ARTICLE_YAML.getBytes(StandardCharsets.UTF_8); + Files.write(sidecar, content); + synchronizer.recordWrittenFile(sidecar, content); + + synchronizer.handleFileCreated(sidecar); + + assertEquals(0, entries().size()); + } + + @Test + void changeToNonHayagrivaContentRemovesItsEntries() throws IOException { + Path file = root.resolve("smith2020.yml"); + Files.writeString(file, ARTICLE_YAML); + openLibrary(); + + Files.writeString(file, """ + jobs: + build: + runs-on: ubuntu-latest + """); + synchronizer.handleFileChanged(file); + + assertEquals(0, entries().size()); + } + + @Test + void createdPdfLinksToExistingSidecarEntry() throws IOException { + Files.writeString(root.resolve("smith2020.yml"), ARTICLE_YAML); + openLibrary(); + BibEntry entry = entries().getFirst(); + assertEquals(List.of(), entry.getFiles()); + + Path pdf = root.resolve("smith2020.pdf"); + Files.createFile(pdf); + synchronizer.handleFileCreated(pdf); + + assertEquals(1, entry.getFiles().size()); + assertEquals("smith2020.pdf", entry.getFiles().getFirst().getLink()); + } + + @Test + void createdPdfWithoutSidecarBecomesStub() throws IOException { + openLibrary(); + Path pdf = root.resolve("interesting-paper.pdf"); + Files.createFile(pdf); + + synchronizer.handleFileCreated(pdf); + + assertEquals(1, entries().size()); + assertEquals(Optional.of("interesting-paper"), entries().getFirst().getField(StandardField.TITLE)); + } + + @Test + void deletedPdfRemovesStubButKeepsSidecarEntry() throws IOException { + Files.writeString(root.resolve("smith2020.yml"), ARTICLE_YAML); + Files.createFile(root.resolve("smith2020.pdf")); + Files.createFile(root.resolve("loose.pdf")); + openLibrary(); + assertEquals(2, entries().size()); + + Files.delete(root.resolve("loose.pdf")); + synchronizer.handleFileDeleted(root.resolve("loose.pdf")); + assertEquals(1, entries().size()); + + Files.delete(root.resolve("smith2020.pdf")); + synchronizer.handleFileDeleted(root.resolve("smith2020.pdf")); + assertEquals(1, entries().size()); + assertTrue(entries().getFirst().getFiles().isEmpty()); + } +} From 597658e46998cc9e4d0125fb4ba05cc42a6ce964 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Tue, 14 Jul 2026 00:08:08 +0200 Subject: [PATCH 2/9] Link CHANGELOG entry to its pull request Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b8a3fdf6a98..49dda935aaf5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv ### Added -- Directory libraries now stay in sync with external file changes: creating, editing, deleting, or renaming `.yml`/`.pdf` files in the opened folder updates the open library live, and renames keep the affected entries (selection and undo history survive). [TODO] +- Directory libraries now stay in sync with external file changes: creating, editing, deleting, or renaming `.yml`/`.pdf` files in the opened folder updates the open library live, and renames keep the affected entries (selection and undo history survive). [#738](https://github.com/JabRef/jabref-koppor/pull/738) - We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear as stub entries titled after the file. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) - The HTTP import endpoint (`POST /libraries/{id}/entries`) now accepts CSL-JSON (`application/vnd.citationstyles.csl+json`), mapping each item to the correct entry type (e.g. conference paper, book chapter, thesis) via the citation-js-based mapping. [#16151](https://github.com/JabRef/jabref/pull/16151) - We added a new "Main" tab to the entry editor showing all fields of an entry in a single scrollable list, with one-click chips for adding optional fields and a free-form box for adding arbitrary fields. Identifiers, files and links, bibliometrics, comments, and meta fields (groups, owner, timestamps, special fields) live in collapsible sections — collapsed when empty — each offering chips for its unset fields. [#12711](https://github.com/JabRef/jabref/issues/12711) From fec84965602c0744145fbf4e82f8013165311c6d Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Tue, 14 Jul 2026 00:14:18 +0200 Subject: [PATCH 3/9] Make org.apache.commons.io transitive in jablib DirectoryMonitor and DirectoryLibrarySynchronizer expose commons-io monitor types in exported-package signatures (flagged by checkModuleDirectivesScope). Co-Authored-By: Claude Fable 5 --- jablib/src/main/java/module-info.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jablib/src/main/java/module-info.java b/jablib/src/main/java/module-info.java index 6ffac1fbae59..7c0f322b886c 100644 --- a/jablib/src/main/java/module-info.java +++ b/jablib/src/main/java/module-info.java @@ -230,7 +230,7 @@ requires transitive com.google.common; requires java.string.similarity; requires transitive org.apache.commons.csv; - requires org.apache.commons.io; + requires transitive org.apache.commons.io; requires org.apache.commons.lang3; requires org.apache.commons.text; // endregion From f2484b7fd0109ed215b394cf9f498fc601ae315b Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Tue, 14 Jul 2026 02:30:18 +0200 Subject: [PATCH 4/9] Merge branch 'open-directory-library' into directory-inbound-sync Externally appearing PDFs also get a generated citation key after insertion, matching the scan behavior. Co-Authored-By: Claude Fable 5 --- .../gui/importer/actions/OpenDirectoryLibraryAction.java | 3 ++- .../directorylibrary/DirectoryLibrarySynchronizer.java | 6 ++++-- .../directorylibrary/DirectoryLibrarySynchronizerTest.java | 3 ++- 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java index 14d8db5fc126..c21ecd55839a 100644 --- a/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java +++ b/jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java @@ -108,7 +108,8 @@ private void showLibraryTab(DirectoryLibraryScanner.ScanResult scanResult) { BibDatabaseContext databaseContext = scanResult.databaseContext(); PdfEntryFactory pdfEntryFactory = new PdfEntryFactory( - preferences.getImportFormatPreferences(), preferences.getFilePreferences()); + preferences.getImportFormatPreferences(), preferences.getFilePreferences(), + preferences.getCitationKeyPatternPreferences()); DirectoryLibrarySynchronizer synchronizer = new DirectoryLibrarySynchronizer( databaseContext, scanResult.catalog(), pdfEntryFactory, UiTaskExecutor::runInJavaFXThread); databaseContext.attachDirectorySynchronizer(synchronizer); diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java index d8982370463d..764a417e691a 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java @@ -358,8 +358,10 @@ private void handlePdfCreated(Path pdf) { return; } BibEntry entry = pdfEntryFactory.createEntry(pdf, root, databaseContext); - modelUpdateMarshaller.accept(() -> - databaseContext.getDatabase().insertEntries(List.of(entry), EntriesEventSource.SHARED)); + modelUpdateMarshaller.accept(() -> { + databaseContext.getDatabase().insertEntries(List.of(entry), EntriesEventSource.SHARED); + pdfEntryFactory.generateCitationKeyIfMissing(entry, databaseContext); + }); } private void handlePdfDeleted(Path pdf) { diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java index 502826e11871..89670860526b 100644 --- a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java @@ -87,7 +87,8 @@ private static PdfEntryFactory offlinePdfEntryFactory() { ImportFormatPreferences importFormatPreferences = mock(ImportFormatPreferences.class, Answers.RETURNS_DEEP_STUBS); when(importFormatPreferences.fieldPreferences().getNonWrappableFields()).thenReturn(FXCollections.emptyObservableList()); when(importFormatPreferences.grobidPreferences()).thenReturn(noGrobid); - return new PdfEntryFactory(importFormatPreferences, mock(FilePreferences.class, Answers.RETURNS_DEEP_STUBS)); + return new PdfEntryFactory(importFormatPreferences, mock(FilePreferences.class, Answers.RETURNS_DEEP_STUBS), + DirectoryLibraryScannerTest.authYearPatternPreferences()); } private List entries() { From 1354d6ded10f913a0c0f498a055edde07d1de59b Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Tue, 14 Jul 2026 04:13:30 +0200 Subject: [PATCH 5/9] Merge branch 'open-directory-library' into directory-inbound-sync Co-Authored-By: Claude Fable 5 --- .../directorylibrary/DirectoryLibrarySynchronizerTest.java | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java index 89670860526b..9f9473b68962 100644 --- a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java @@ -16,6 +16,8 @@ import org.jabref.logic.FilePreferences; import org.jabref.logic.importer.ImportFormatPreferences; +import org.jabref.logic.importer.fetcher.CrossRef; +import org.jabref.logic.importer.fetcher.DoiFetcher; import org.jabref.logic.importer.util.GrobidPreferences; import org.jabref.model.database.BibDatabaseContext; import org.jabref.model.entry.BibEntry; @@ -88,7 +90,7 @@ private static PdfEntryFactory offlinePdfEntryFactory() { when(importFormatPreferences.fieldPreferences().getNonWrappableFields()).thenReturn(FXCollections.emptyObservableList()); when(importFormatPreferences.grobidPreferences()).thenReturn(noGrobid); return new PdfEntryFactory(importFormatPreferences, mock(FilePreferences.class, Answers.RETURNS_DEEP_STUBS), - DirectoryLibraryScannerTest.authYearPatternPreferences()); + DirectoryLibraryScannerTest.authYearPatternPreferences(), mock(CrossRef.class), mock(DoiFetcher.class)); } private List entries() { From e58d1412565f8f8ec6674456bf44269ace64a1b2 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Fri, 17 Jul 2026 01:49:57 +0200 Subject: [PATCH 6/9] Watch Markdown sidecars for inbound synchronization External creation, modification, deletion, and renaming of .md files with Hayagriva frontmatter now flows into the open directory library like the plain .yml sidecars; the notes body maps to the entry's comment fields. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Q5Y5GTihE9rZxKrvPqnHV4 --- CHANGELOG.md | 2 +- docs/requirements/directory-library.md | 4 +- .../DirectoryLibrarySynchronizer.java | 42 ++++++++++++------- .../DirectoryLibrarySynchronizerTest.java | 42 +++++++++++++++++++ 4 files changed, 72 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3012cb5c8a94..7eca0cf479e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv ### Added -- Directory libraries now stay in sync with external file changes: creating, editing, deleting, or renaming `.yml`/`.pdf` files in the opened folder updates the open library live, and renames keep the affected entries (selection and undo history survive). [#738](https://github.com/JabRef/jabref-koppor/pull/738) +- Directory libraries now stay in sync with external file changes: creating, editing, deleting, or renaming `.yml`/`.md`/`.pdf` files in the opened folder updates the open library live, and renames keep the affected entries (selection and undo history survive). [#738](https://github.com/JabRef/jabref-koppor/pull/738) - We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear as stub entries titled after the file. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) - We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear as entries with metadata extracted from the PDF itself (falling back to the file name). Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) - We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear immediately and are enriched in the background with metadata extracted from the PDF itself and a generated citation key. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) diff --git a/docs/requirements/directory-library.md b/docs/requirements/directory-library.md index afd3c170c0b7..d93e10f9d80e 100644 --- a/docs/requirements/directory-library.md +++ b/docs/requirements/directory-library.md @@ -33,10 +33,10 @@ the last-opened list and routed back through the directory-library opener. Needs: impl ## External file changes appear live in an open directory library -`req~directory-library.inbound-sync~1` +`req~directory-library.inbound-sync~2` While a directory library is open, external creation, modification, deletion, and renaming of -`.yml`/`.yaml`/`.pdf` files under its root must be reflected in the open library. Changed +`.yml`/`.yaml`/`.md`/`.pdf` files under its root must be reflected in the open library. Changed entries are updated in place (the entry identity is preserved), renames are detected via a grace window over the monitor's delete + create events and keep the affected entries, and files written by JabRef itself are recognized by fingerprint and not re-imported. All resulting diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java index 764a417e691a..f14d38e8aef6 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java @@ -64,7 +64,10 @@ /// grace period spanning two poll cycles: a create whose parsed entries equal a staged /// deletion's entries is treated as a move (the [BibEntry] instances survive, preserving /// selection and undo history); only unmatched deletions are committed. -// [impl->req~directory-library.inbound-sync~1] +/// +/// Sidecars come in two forms (see [MarkdownSidecar]): plain Hayagriva `.yml`/`.yaml` files and +/// Markdown `.md` files whose Hayagriva frontmatter carries the data; both are watched alike. +// [impl->req~directory-library.inbound-sync~2] @NullMarked public class DirectoryLibrarySynchronizer implements FileAlterationListener { @@ -74,7 +77,7 @@ public class DirectoryLibrarySynchronizer implements FileAlterationListener { /// in the poll cycle after its delete event. private static final Duration RENAME_GRACE = Duration.ofMillis(2500); - private static final Set YAML_EXTENSIONS = Set.of("yml", "yaml"); + private static final Set SIDECAR_EXTENSIONS = Set.of("yml", "yaml", MarkdownSidecar.MARKDOWN_EXTENSION); private static final String PDF_EXTENSION = "pdf"; private final BibDatabaseContext databaseContext; @@ -84,6 +87,7 @@ public class DirectoryLibrarySynchronizer implements FileAlterationListener { private final Consumer modelUpdateMarshaller; private final Clock clock; private final HayagrivaImporter importer = new HayagrivaImporter(); + private final MarkdownSidecar markdownSidecar = new MarkdownSidecar(); private final ScheduledExecutorService syncExecutor; private final Map stagedDeletions = new HashMap<>(); @@ -127,6 +131,7 @@ public void startWatching(DirectoryMonitor monitor) { FileFilterUtils.directoryFileFilter(), FileFilterUtils.suffixFileFilter(".yml", IOCase.INSENSITIVE), FileFilterUtils.suffixFileFilter(".yaml", IOCase.INSENSITIVE), + FileFilterUtils.suffixFileFilter(".md", IOCase.INSENSITIVE), FileFilterUtils.suffixFileFilter(".pdf", IOCase.INSENSITIVE)); IOFileFilter notHidden = FileFilterUtils.notFileFilter(FileFilterUtils.prefixFileFilter(".")); observer = FileAlterationObserver.builder() @@ -195,7 +200,7 @@ public void onStop(FileAlterationObserver observer) { synchronized void handleFileCreated(Path file) { commitExpiredStagedDeletions(); - if (isYaml(file)) { + if (isSidecar(file)) { if (consumeSelfEcho(file)) { return; } @@ -207,7 +212,7 @@ synchronized void handleFileCreated(Path file) { synchronized void handleFileChanged(Path file) { commitExpiredStagedDeletions(); - if (!isYaml(file) || consumeSelfEcho(file)) { + if (!isSidecar(file) || consumeSelfEcho(file)) { return; } List knownEntries = entriesOf(file); @@ -215,8 +220,8 @@ synchronized void handleFileChanged(Path file) { importFile(file); return; } - if (!looksLikeHayagriva(file)) { - // The file stopped being a Hayagriva file (e.g. replaced by unrelated YAML) + if (!looksLikeSidecar(file)) { + // The file stopped being a sidecar (e.g. replaced by unrelated YAML or Markdown) removeEntries(knownEntries, file); return; } @@ -230,7 +235,7 @@ synchronized void handleFileChanged(Path file) { synchronized void handleFileDeleted(Path file) { commitExpiredStagedDeletions(); - if (isYaml(file)) { + if (isSidecar(file)) { List entries = entriesOf(file); if (entries.isEmpty()) { return; @@ -260,7 +265,7 @@ private void importFile(Path file) { handleFileChanged(file); return; } - if (!looksLikeHayagriva(file)) { + if (!looksLikeSidecar(file)) { return; } Optional> parsed = parse(file); @@ -421,7 +426,9 @@ private static boolean entriesMatch(List staged, List parsed private Optional> parse(Path file) { try { - ParserResult parserResult = importer.importDatabase(file); + ParserResult parserResult = MarkdownSidecar.hasMarkdownExtension(file) + ? markdownSidecar.read(file) + : importer.importDatabase(file); if (parserResult.isInvalid()) { return Optional.empty(); } @@ -432,9 +439,14 @@ private Optional> parse(Path file) { } } - private boolean looksLikeHayagriva(Path file) { - try (BufferedReader reader = Files.newBufferedReader(file, StandardCharsets.UTF_8)) { - return importer.isRecognizedFormat(reader); + private boolean looksLikeSidecar(Path file) { + try { + if (MarkdownSidecar.hasMarkdownExtension(file)) { + return markdownSidecar.looksLikeSidecar(file); + } + try (BufferedReader reader = Files.newBufferedReader(file, StandardCharsets.UTF_8)) { + return importer.isRecognizedFormat(reader); + } } catch (IOException e) { LOGGER.warn("Could not read {}", file, e); return false; @@ -447,7 +459,7 @@ private Optional findSidecarEntry(Path pdf) { return Optional.empty(); } String baseName = FileUtil.getBaseName(pdf); - for (String extension : YAML_EXTENSIONS) { + for (String extension : SIDECAR_EXTENSIONS) { List entries = entriesOf(parent.resolve(baseName + "." + extension)); if (!entries.isEmpty()) { return Optional.of(entries.getFirst()); @@ -491,8 +503,8 @@ private static String hash(byte[] content) { } } - private static boolean isYaml(Path file) { - return YAML_EXTENSIONS.contains(FileUtil.getFileExtension(file).orElse("").toLowerCase(Locale.ROOT)); + private static boolean isSidecar(Path file) { + return SIDECAR_EXTENSIONS.contains(FileUtil.getFileExtension(file).orElse("").toLowerCase(Locale.ROOT)); } private static boolean isPdf(Path file) { diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java index 9f9473b68962..dec5ca9c9700 100644 --- a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java @@ -43,6 +43,19 @@ class DirectoryLibrarySynchronizerTest { note: first version """; + private static final String MARKDOWN_SIDECAR = """ + --- + smith2020: + type: article + title: A Test Article + author: Smith, Jane + --- + + # Notes + + Shared comment text. + """; + /// Deterministic clock for the rename grace window. private static final class SteppingClock extends Clock { private Instant now = Instant.parse("2026-07-13T12:00:00Z"); @@ -130,6 +143,35 @@ void externalChangeUpdatesTheSameEntryInstance() throws IOException { assertEquals(1, entry.getFiles().size()); } + @Test + void externallyCreatedMarkdownSidecarAddsEntryWithComments() throws IOException { + openLibrary(); + Path sidecar = root.resolve("smith2020.md"); + Files.writeString(sidecar, MARKDOWN_SIDECAR); + + synchronizer.handleFileCreated(sidecar); + + assertEquals(1, entries().size()); + BibEntry added = entries().getFirst(); + assertEquals(Optional.of("smith2020"), added.getCitationKey()); + assertEquals(Optional.of("Shared comment text."), added.getField(StandardField.COMMENT)); + } + + @Test + void externalMarkdownChangeUpdatesCommentOnTheSameEntryInstance() throws IOException { + Path sidecar = root.resolve("smith2020.md"); + Files.writeString(sidecar, MARKDOWN_SIDECAR); + openLibrary(); + BibEntry entry = entries().getFirst(); + + Files.writeString(sidecar, MARKDOWN_SIDECAR.replace("Shared comment text.", "Updated comment text.")); + synchronizer.handleFileChanged(sidecar); + + assertEquals(1, entries().size()); + assertSame(entry, entries().getFirst()); + assertEquals(Optional.of("Updated comment text."), entry.getField(StandardField.COMMENT)); + } + @Test void externalChangeAddsAndRemovesEntriesOfMultiEntryFile() throws IOException { Path file = root.resolve("collection.yml"); From 983cdb5c317dbce2428758dd21d40556cca90b6f Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Thu, 3 Sep 2026 04:20:37 +0200 Subject: [PATCH 7/9] Harden the inbound directory synchronization A deletion undone within the grace window no longer removes live entries, renames of sidecars with a paired PDF keep the entry instances, an editor polled mid-write only stages the entries instead of dropping them, and events arriving during shutdown are discarded instead of failing the shared monitor thread. Sidecar precedence is deterministic; the baseline snapshot runs off the caller's thread. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01Vr3E1Gg5DRU4LQDDVnhPys --- CHANGELOG.md | 5 - .../DirectoryLibraryCatalog.java | 19 +- .../DirectoryLibrarySynchronizer.java | 192 +++++++++--------- .../jabref/logic/util/DirectoryMonitor.java | 7 +- .../DirectoryLibrarySynchronizerTest.java | 75 ++++++- 5 files changed, 177 insertions(+), 121 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ad1b37ace190..aeee9bea5c32 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,11 +12,6 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv ### Added - Directory libraries now stay in sync with external file changes: creating, editing, deleting, or renaming `.yml`/`.md`/`.pdf` files in the opened folder updates the open library live, and renames keep the affected entries (selection and undo history survive). [#738](https://github.com/JabRef/jabref-koppor/pull/738) -- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear as stub entries titled after the file. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) -- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear as entries with metadata extracted from the PDF itself (falling back to the file name). Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) -- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear immediately and are enriched in the background with metadata extracted from the PDF itself and a generated citation key. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) -- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear immediately and are enriched in the background with metadata extracted from the PDF itself and a generated citation key; a missing DOI is looked up online and its metadata fills the remaining empty fields. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) -- We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files and PDFs found in the folder tree. A PDF next to a `.yml` sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear immediately and are enriched in the background with metadata extracted from the PDF itself and a generated citation key; a missing DOI is looked up online and its metadata fills the remaining empty fields. Directory libraries that were open on shutdown are reopened on the next start. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) - We added "Open folder as library" (File menu): JabRef opens a directory as a library, filling the main table from the Hayagriva `.yml` files, the Markdown notes files with a Hayagriva YAML frontmatter (`.md`; the notes body maps to the entry's comment fields), and the PDFs found in the folder tree. A PDF next to a sidecar of the same name is linked to the sidecar's entry; PDFs without a sidecar appear immediately and are enriched in the background with metadata extracted from the PDF itself and a generated citation key; a missing DOI is looked up online and its metadata fills the remaining empty fields. Directory libraries that were open on shutdown are reopened on the next start. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737) - We added a "Commit and push" button which allows to commit and then push in one go for Git operations. [#16339](https://github.com/JabRef/jabref/issues/16339) - We added the option to close and reopen the PDF preview in the unlinked files dialog. [#16159](https://github.com/JabRef/jabref/issues/16159) diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryCatalog.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryCatalog.java index fcb1f0dfc104..53a0b6bf4ab5 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryCatalog.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryCatalog.java @@ -43,24 +43,15 @@ public List entryIdsIn(Path yamlFile) { /// Re-homes all entries of `oldFile` to `newFile` (a rename/move on disk). public void relocateFile(Path oldFile, Path newFile) { - List entryIds = entryIdsByFile.remove(oldFile); - if (entryIds == null) { - return; - } - entryIdsByFile.put(newFile, entryIds); - entryIds.forEach(entryId -> { - EntrySource source = sourceByEntryId.get(entryId); - if (source != null) { - sourceByEntryId.put(entryId, new EntrySource(newFile, source.hayagrivaKey())); - } + Optional.ofNullable(entryIdsByFile.remove(oldFile)).ifPresent(entryIds -> { + entryIdsByFile.put(newFile, entryIds); + entryIds.forEach(entryId -> sourceByEntryId.computeIfPresent(entryId, + (_, source) -> new EntrySource(newFile, source.hayagrivaKey()))); }); } /// Forgets all entries of the given file (deleted on disk or re-registered afterwards). public void removeFile(Path yamlFile) { - List entryIds = entryIdsByFile.remove(yamlFile); - if (entryIds != null) { - entryIds.forEach(sourceByEntryId::remove); - } + Optional.ofNullable(entryIdsByFile.remove(yamlFile)).ifPresent(entryIds -> entryIds.forEach(sourceByEntryId::remove)); } } diff --git a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java index f14d38e8aef6..f6573a467afb 100644 --- a/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java +++ b/jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizer.java @@ -20,11 +20,13 @@ import java.util.Map; import java.util.Optional; import java.util.SequencedMap; -import java.util.Set; -import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.ScheduledThreadPoolExecutor; +import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.TimeUnit; import java.util.function.Consumer; +import java.util.stream.Collectors; +import java.util.stream.IntStream; import org.jabref.logic.bibtex.FileFieldWriter; import org.jabref.logic.importer.ParserResult; @@ -36,6 +38,7 @@ import org.jabref.model.entry.BibEntry; import org.jabref.model.entry.LinkedFile; import org.jabref.model.entry.event.EntriesEventSource; +import org.jabref.model.entry.field.Field; import org.jabref.model.entry.field.StandardField; import org.apache.commons.io.IOCase; @@ -73,11 +76,12 @@ public class DirectoryLibrarySynchronizer implements FileAlterationListener { private static final Logger LOGGER = LoggerFactory.getLogger(DirectoryLibrarySynchronizer.class); - /// Two poll cycles of [DirectoryMonitor] (1 s each), so a rename's create event can arrive - /// in the poll cycle after its delete event. - private static final Duration RENAME_GRACE = Duration.ofMillis(2500); + /// Two poll cycles of [DirectoryMonitor], so a rename's create event can arrive in the poll + /// cycle after its delete event. + private static final Duration RENAME_GRACE = DirectoryMonitor.POLL_INTERVAL.multipliedBy(2).plusMillis(500); - private static final Set SIDECAR_EXTENSIONS = Set.of("yml", "yaml", MarkdownSidecar.MARKDOWN_EXTENSION); + /// In precedence order when several sidecars share a base name. + private static final List SIDECAR_EXTENSIONS = List.of("yml", "yaml", MarkdownSidecar.MARKDOWN_EXTENSION); private static final String PDF_EXTENSION = "pdf"; private final BibDatabaseContext databaseContext; @@ -93,12 +97,14 @@ public class DirectoryLibrarySynchronizer implements FileAlterationListener { private final Map stagedDeletions = new HashMap<>(); private final Map lastWrittenFingerprints = new HashMap<>(); - private @Nullable FileAlterationObserver observer; - private @Nullable DirectoryMonitor directoryMonitor; + private @Nullable Watch watch; private record StagedDeletion(List entries, Instant expiry) { } + private record Watch(DirectoryMonitor monitor, FileAlterationObserver observer) { + } + public DirectoryLibrarySynchronizer(BibDatabaseContext databaseContext, DirectoryLibraryCatalog catalog, PdfEntryFactory pdfEntryFactory, @@ -118,15 +124,16 @@ public DirectoryLibrarySynchronizer(BibDatabaseContext databaseContext, () -> new IllegalArgumentException("Context is not a directory library")); this.modelUpdateMarshaller = modelUpdateMarshaller; this.clock = clock; - this.syncExecutor = Executors.newSingleThreadScheduledExecutor(runnable -> { - Thread thread = new Thread(runnable, "directory-sync"); - thread.setDaemon(true); - return thread; - }); + // A dedicated single thread (not BackgroundTask: events must be serialized and writes + // debounced). Events polled while this synchronizer shuts down are dropped instead of + // throwing into the shared monitor thread. + ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(1, + Thread.ofPlatform().name("directory-sync").daemon(true).factory()); + executor.setRejectedExecutionHandler(new ThreadPoolExecutor.DiscardPolicy()); + this.syncExecutor = executor; } public void startWatching(DirectoryMonitor monitor) { - this.directoryMonitor = monitor; IOFileFilter relevantFiles = FileFilterUtils.or( FileFilterUtils.directoryFileFilter(), FileFilterUtils.suffixFileFilter(".yml", IOCase.INSENSITIVE), @@ -134,21 +141,23 @@ public void startWatching(DirectoryMonitor monitor) { FileFilterUtils.suffixFileFilter(".md", IOCase.INSENSITIVE), FileFilterUtils.suffixFileFilter(".pdf", IOCase.INSENSITIVE)); IOFileFilter notHidden = FileFilterUtils.notFileFilter(FileFilterUtils.prefixFileFilter(".")); - observer = FileAlterationObserver.builder() - .setRootEntry(new FileEntry(root.toFile())) - .setFileFilter(FileFilterUtils.and(notHidden, relevantFiles)) - .getUnchecked(); + FileAlterationObserver observer = FileAlterationObserver.builder() + .setRootEntry(new FileEntry(root.toFile())) + .setFileFilter(FileFilterUtils.and(notHidden, relevantFiles)) + .getUnchecked(); + watch = new Watch(monitor, observer); // The monitor is already running and never initializes late-joining observers, so the // first poll would report every existing file as created. Checking once without any - // listener attached takes the baseline snapshot silently. - observer.checkAndNotify(); - monitor.addObserver(observer, this); + // listener attached takes the baseline snapshot silently — off the caller's thread, + // since it walks the whole tree. + syncExecutor.execute(() -> { + observer.checkAndNotify(); + monitor.addObserver(observer, this); + }); } public void shutdown() { - if (observer != null && directoryMonitor != null) { - directoryMonitor.removeObserver(observer); - } + Optional.ofNullable(watch).ifPresent(active -> active.monitor().removeObserver(active.observer())); syncExecutor.shutdown(); } @@ -221,16 +230,16 @@ synchronized void handleFileChanged(Path file) { return; } if (!looksLikeSidecar(file)) { - // The file stopped being a sidecar (e.g. replaced by unrelated YAML or Markdown) - removeEntries(knownEntries, file); + // The file stopped being a sidecar — or an editor that truncates and rewrites was + // polled mid-write, so the entries are only staged: a complete sidecar arriving + // within the grace window keeps them + stageDeletion(file, knownEntries); return; } - Optional> parsed = parse(file); - if (parsed.isEmpty()) { - LOGGER.warn("Not applying changes of unparseable Hayagriva file {}", file); - return; - } - applyChangedFile(file, knownEntries, parsed.get()); + parse(file).ifPresentOrElse(parsedEntries -> { + stagedDeletions.remove(file); + applyChangedFile(file, knownEntries, parsedEntries); + }, () -> LOGGER.warn("Not applying changes of unparseable Hayagriva file {}", file)); } synchronized void handleFileDeleted(Path file) { @@ -240,14 +249,17 @@ synchronized void handleFileDeleted(Path file) { if (entries.isEmpty()) { return; } - stagedDeletions.put(file, new StagedDeletion(entries, clock.instant().plus(RENAME_GRACE))); - syncExecutor.schedule(this::commitExpiredStagedDeletions, - RENAME_GRACE.toMillis() + 100, TimeUnit.MILLISECONDS); + stageDeletion(file, entries); } else if (isPdf(file)) { handlePdfDeleted(file); } } + private void stageDeletion(Path file, List entries) { + stagedDeletions.put(file, new StagedDeletion(entries, clock.instant().plus(RENAME_GRACE))); + syncExecutor.schedule(this::commitExpiredStagedDeletions, RENAME_GRACE.toMillis() + 100, TimeUnit.MILLISECONDS); + } + synchronized void commitExpiredStagedDeletions() { Instant now = clock.instant(); List> expired = stagedDeletions.entrySet().stream() @@ -261,32 +273,34 @@ synchronized void commitExpiredStagedDeletions() { private void importFile(Path file) { if (!entriesOf(file).isEmpty()) { - // Already known (e.g. a create event for a file the scan covered) — diff instead + // Already known: a create event for a file the scan covered, or a deletion undone + // within the grace window — diff instead of importing twice + stagedDeletions.remove(file); handleFileChanged(file); return; } if (!looksLikeSidecar(file)) { return; } - Optional> parsed = parse(file); - if (parsed.isEmpty() || parsed.get().isEmpty()) { + List newEntries = parse(file).orElse(List.of()); + if (newEntries.isEmpty()) { return; } - List newEntries = parsed.get(); // A staged deletion with equal content is this file being moved, not new content - Optional movedFrom = stagedDeletions.entrySet().stream() - .filter(staged -> entriesMatch(staged.getValue().entries(), newEntries)) - .map(Map.Entry::getKey) - .findFirst(); - if (movedFrom.isPresent()) { - stagedDeletions.remove(movedFrom.get()); - catalog.relocateFile(movedFrom.get(), file); - LOGGER.debug("Detected move {} -> {}", movedFrom.get(), file); - return; - } - - newEntries.forEach(entry -> catalog.register(entry, file, entry.getCitationKey().orElse(""))); + stagedDeletions.entrySet().stream() + .filter(staged -> entriesMatch(staged.getValue().entries(), newEntries)) + .map(Map.Entry::getKey) + .findFirst() + .ifPresentOrElse(movedFrom -> { + stagedDeletions.remove(movedFrom); + catalog.relocateFile(movedFrom, file); + LOGGER.debug("Detected move {} -> {}", movedFrom, file); + }, () -> insertNewEntries(file, newEntries)); + } + + private void insertNewEntries(Path file, List newEntries) { + newEntries.forEach(entry -> catalog.register(entry, file, entry.getCitationKey().orElseThrow())); // Safe without event source: the entry is not yet inserted, so no listeners see this findPairedPdf(file).ifPresent(pdf -> newEntries.getFirst() .addFile(new LinkedFile("", root.relativize(pdf), StandardFileType.PDF.getName()))); @@ -302,15 +316,13 @@ private void applyChangedFile(Path file, List knownEntries, List toRemove = new ArrayList<>(); List fieldUpdates = new ArrayList<>(); - parsedByKey.forEach((key, parsedEntry) -> { - BibEntry knownEntry = knownByKey.get(key); - if (knownEntry == null) { - catalog.register(parsedEntry, file, key); - toInsert.add(parsedEntry); - } else { - fieldUpdates.add(() -> copyContent(parsedEntry, knownEntry)); - } - }); + parsedByKey.forEach((key, parsedEntry) -> + Optional.ofNullable(knownByKey.get(key)).ifPresentOrElse( + knownEntry -> fieldUpdates.add(() -> copyContent(parsedEntry, knownEntry)), + () -> { + catalog.register(parsedEntry, file, key); + toInsert.add(parsedEntry); + })); knownByKey.forEach((key, knownEntry) -> { if (!parsedByKey.containsKey(key)) { toRemove.add(knownEntry); @@ -413,15 +425,19 @@ private static SequencedMap byCitationKey(List entri } private static boolean entriesMatch(List staged, List parsed) { - if (staged.size() != parsed.size()) { - return false; - } - for (int i = 0; i < staged.size(); i++) { - if (!staged.get(i).equals(parsed.get(i))) { - return false; - } - } - return true; + return staged.size() == parsed.size() + && IntStream.range(0, staged.size()).allMatch(i -> sameContent(staged.get(i), parsed.get(i))); + } + + /// Live entries carry the PDF link this synchronizer maintains; freshly parsed ones do not. + private static boolean sameContent(BibEntry live, BibEntry parsed) { + return live.getType().equals(parsed.getType()) && fieldsWithoutFile(live).equals(fieldsWithoutFile(parsed)); + } + + private static Map fieldsWithoutFile(BibEntry entry) { + return entry.getFieldMap().entrySet().stream() + .filter(field -> StandardField.FILE != field.getKey()) + .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)); } private Optional> parse(Path file) { @@ -454,45 +470,33 @@ private boolean looksLikeSidecar(Path file) { } private Optional findSidecarEntry(Path pdf) { - Path parent = pdf.getParent(); - if (parent == null) { - return Optional.empty(); - } String baseName = FileUtil.getBaseName(pdf); - for (String extension : SIDECAR_EXTENSIONS) { - List entries = entriesOf(parent.resolve(baseName + "." + extension)); - if (!entries.isEmpty()) { - return Optional.of(entries.getFirst()); - } - } - return Optional.empty(); + return SIDECAR_EXTENSIONS.stream() + .map(extension -> entriesOf(pdf.resolveSibling(baseName + "." + extension))) + .filter(entries -> !entries.isEmpty()) + .map(List::getFirst) + .findFirst(); } private Optional findPairedPdf(Path yamlFile) { - Path parent = yamlFile.getParent(); - if (parent == null) { - return Optional.empty(); - } - Path pdf = parent.resolve(FileUtil.getBaseName(yamlFile) + ".pdf"); - return Files.exists(pdf) ? Optional.of(pdf) : Optional.empty(); + return Optional.of(yamlFile.resolveSibling(FileUtil.getBaseName(yamlFile) + ".pdf")).filter(Files::exists); } private boolean consumeSelfEcho(Path file) { Path normalized = file.toAbsolutePath().normalize(); - String recorded = lastWrittenFingerprints.get(normalized); - if (recorded == null) { + if (!lastWrittenFingerprints.containsKey(normalized)) { return false; } + return currentHash(file).map(current -> lastWrittenFingerprints.remove(normalized, current)).orElse(false); + } + + private static Optional currentHash(Path file) { try { - String current = hash(Files.readAllBytes(file)); - if (recorded.equals(current)) { - lastWrittenFingerprints.remove(normalized); - return true; - } + return Optional.of(hash(Files.readAllBytes(file))); } catch (IOException e) { LOGGER.debug("Could not fingerprint {}", file, e); + return Optional.empty(); } - return false; } private static String hash(byte[] content) { diff --git a/jablib/src/main/java/org/jabref/logic/util/DirectoryMonitor.java b/jablib/src/main/java/org/jabref/logic/util/DirectoryMonitor.java index 75a0d66dec6a..6a3f14edc3bf 100644 --- a/jablib/src/main/java/org/jabref/logic/util/DirectoryMonitor.java +++ b/jablib/src/main/java/org/jabref/logic/util/DirectoryMonitor.java @@ -1,5 +1,7 @@ package org.jabref.logic.util; +import java.time.Duration; + import org.apache.commons.io.monitor.FileAlterationListener; import org.apache.commons.io.monitor.FileAlterationMonitor; import org.apache.commons.io.monitor.FileAlterationObserver; @@ -8,13 +10,14 @@ public class DirectoryMonitor { + public static final Duration POLL_INTERVAL = Duration.ofSeconds(1); + private static final Logger LOGGER = LoggerFactory.getLogger(DirectoryMonitor.class); - private static final int POLL_INTERVAL = 1000; private final FileAlterationMonitor monitor; public DirectoryMonitor() { - monitor = new FileAlterationMonitor(POLL_INTERVAL); + monitor = new FileAlterationMonitor(POLL_INTERVAL.toMillis()); start(); } diff --git a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java index dec5ca9c9700..a2616e67f92f 100644 --- a/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java +++ b/jablib/src/test/java/org/jabref/logic/directorylibrary/DirectoryLibrarySynchronizerTest.java @@ -23,13 +23,13 @@ import org.jabref.model.entry.BibEntry; import org.jabref.model.entry.field.StandardField; +import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; import org.mockito.Answers; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertSame; -import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; @@ -106,6 +106,11 @@ private static PdfEntryFactory offlinePdfEntryFactory() { DirectoryLibraryScannerTest.authYearPatternPreferences(), mock(CrossRef.class), mock(DoiFetcher.class)); } + @AfterEach + void shutdown() { + synchronizer.shutdown(); + } + private List entries() { return context.getDatabase().getEntries(); } @@ -229,8 +234,44 @@ void renameIsDetectedAsMoveAndPreservesEntryInstance() throws IOException { clock.advance(Duration.ofSeconds(3)); synchronizer.commitExpiredStagedDeletions(); - assertEquals(1, entries().size()); - assertSame(entry, entries().getFirst()); + assertEquals(List.of(entry), entries()); + } + + @Test + void renameOfSidecarWithPairedPdfIsDetectedAsMove() throws IOException { + Path oldFile = root.resolve("smith2020.yml"); + Files.writeString(oldFile, ARTICLE_YAML); + Files.createFile(root.resolve("smith2020.pdf")); + openLibrary(); + BibEntry entry = entries().getFirst(); + + Path newFile = root.resolve("renamed.yml"); + Files.move(oldFile, newFile); + synchronizer.handleFileDeleted(oldFile); + synchronizer.handleFileCreated(newFile); + clock.advance(Duration.ofSeconds(3)); + synchronizer.commitExpiredStagedDeletions(); + + assertEquals(List.of(entry), entries()); + assertEquals("smith2020.pdf", entry.getFiles().getFirst().getLink()); + } + + @Test + void deletionUndoneWithinGraceWindowKeepsEntry() throws IOException { + Path sidecar = root.resolve("smith2020.yml"); + Files.writeString(sidecar, ARTICLE_YAML); + openLibrary(); + BibEntry entry = entries().getFirst(); + + Files.delete(sidecar); + synchronizer.handleFileDeleted(sidecar); + Files.writeString(sidecar, ARTICLE_YAML.replace("first version", "restored version")); + synchronizer.handleFileCreated(sidecar); + clock.advance(Duration.ofSeconds(3)); + synchronizer.commitExpiredStagedDeletions(); + + assertEquals(List.of(entry), entries()); + assertEquals(Optional.of("restored version"), entry.getField(StandardField.NOTE)); } @Test @@ -247,7 +288,7 @@ void selfWrittenFileIsNotReimported() throws IOException { } @Test - void changeToNonHayagrivaContentRemovesItsEntries() throws IOException { + void changeToNonHayagrivaContentRemovesItsEntriesAfterGraceWindow() throws IOException { Path file = root.resolve("smith2020.yml"); Files.writeString(file, ARTICLE_YAML); openLibrary(); @@ -258,8 +299,30 @@ void changeToNonHayagrivaContentRemovesItsEntries() throws IOException { runs-on: ubuntu-latest """); synchronizer.handleFileChanged(file); + assertEquals(1, entries().size()); - assertEquals(0, entries().size()); + clock.advance(Duration.ofSeconds(3)); + synchronizer.commitExpiredStagedDeletions(); + assertEquals(List.of(), entries()); + } + + /// Editors that truncate and rewrite can be polled mid-write. + @Test + void sidecarCompletedWithinGraceWindowKeepsEntry() throws IOException { + Path file = root.resolve("smith2020.yml"); + Files.writeString(file, ARTICLE_YAML); + openLibrary(); + BibEntry entry = entries().getFirst(); + + Files.writeString(file, "smith2020:\n"); + synchronizer.handleFileChanged(file); + Files.writeString(file, ARTICLE_YAML.replace("first version", "second version")); + synchronizer.handleFileChanged(file); + clock.advance(Duration.ofSeconds(3)); + synchronizer.commitExpiredStagedDeletions(); + + assertEquals(List.of(entry), entries()); + assertEquals(Optional.of("second version"), entry.getField(StandardField.NOTE)); } @Test @@ -304,6 +367,6 @@ void deletedPdfRemovesStubButKeepsSidecarEntry() throws IOException { Files.delete(root.resolve("smith2020.pdf")); synchronizer.handleFileDeleted(root.resolve("smith2020.pdf")); assertEquals(1, entries().size()); - assertTrue(entries().getFirst().getFiles().isEmpty()); + assertEquals(List.of(), entries().getFirst().getFiles()); } } From 084c0cc67a9dddb0b844c2807db4734c0f027042 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Mon, 7 Sep 2026 00:16:02 +0200 Subject: [PATCH 8/9] Apply IntelliJ formatter after upstream resync Fixes the CI format check. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DDcHNMt9fPWnpYaHheFvry --- .../org/jabref/logic/exporter/HayagrivaEntryWriter.java | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/jablib/src/main/java/org/jabref/logic/exporter/HayagrivaEntryWriter.java b/jablib/src/main/java/org/jabref/logic/exporter/HayagrivaEntryWriter.java index 46f439a2fd7f..271ac32dd295 100644 --- a/jablib/src/main/java/org/jabref/logic/exporter/HayagrivaEntryWriter.java +++ b/jablib/src/main/java/org/jabref/logic/exporter/HayagrivaEntryWriter.java @@ -313,9 +313,12 @@ private ObjectNode parentNode(String type, String title) { private String booktitleParentType(EntryType entryType) { return switch (entryType) { - case StandardEntryType.InProceedings -> "proceedings"; - case StandardEntryType.InBook -> "book"; - default -> "anthology"; + case StandardEntryType.InProceedings -> + "proceedings"; + case StandardEntryType.InBook -> + "book"; + default -> + "anthology"; }; } From ae191b0718a7e3abae6c16a2d3958c91e35b2840 Mon Sep 17 00:00:00 2001 From: Oliver Kopp Date: Mon, 7 Sep 2026 11:13:46 +0200 Subject: [PATCH 9/9] Renumber the directory-as-library ADR to 0072 Upstream added a 0071 ADR after this stack claimed the number, so the MADR duplicate-ID check failed. Co-Authored-By: Claude Opus 4.8 --- ....md => 0072-directory-as-library-with-hayagriva-sidecars.md} | 2 +- docs/requirements/directory-library.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) rename docs/decisions/{0071-directory-as-library-with-hayagriva-sidecars.md => 0072-directory-as-library-with-hayagriva-sidecars.md} (99%) diff --git a/docs/decisions/0071-directory-as-library-with-hayagriva-sidecars.md b/docs/decisions/0072-directory-as-library-with-hayagriva-sidecars.md similarity index 99% rename from docs/decisions/0071-directory-as-library-with-hayagriva-sidecars.md rename to docs/decisions/0072-directory-as-library-with-hayagriva-sidecars.md index 5229eadcaf1f..79ff20e89107 100644 --- a/docs/decisions/0071-directory-as-library-with-hayagriva-sidecars.md +++ b/docs/decisions/0072-directory-as-library-with-hayagriva-sidecars.md @@ -1,5 +1,5 @@ --- -nav_order: 0071 +nav_order: 0072 parent: Decision Records --- diff --git a/docs/requirements/directory-library.md b/docs/requirements/directory-library.md index 1a31fe9a5cbf..3a3e0f6b1445 100644 --- a/docs/requirements/directory-library.md +++ b/docs/requirements/directory-library.md @@ -19,7 +19,7 @@ online and the metadata behind it fills only the fields the PDF did not provide. and `.md` files without a Hayagriva frontmatter are skipped; unparseable Hayagriva files are reported as warnings without aborting the scan. Scanning must not write or modify any file in the directory. -See [ADR 71](../decisions/0071-directory-as-library-with-hayagriva-sidecars.md) for more details. +See [ADR 72](../decisions/0072-directory-as-library-with-hayagriva-sidecars.md) for more details. Needs: impl