Write-back to Hayagriva sidecars (directory as library, phase 3)

CHANGELOG.md

@@ -11,6 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv
 
 ### Added
 
+- Directory libraries now save into their sidecar files: edits are written back automatically (debounced until typing pauses; Ctrl+S forces the write and no longer creates a `.bib`), the first edit of a PDF-only entry creates a Markdown sidecar (`X.md` with the Hayagriva data as frontmatter and the comment fields as notes body), renaming a citation key renames the YAML key, and deleting an entry removes it from its file (the file is trashed once empty, the PDF stays). Hand-written content that JabRef does not understand survives rewrites. [#739](https://github.com/JabRef/jabref-koppor/pull/739)
 - Directory libraries now stay in sync with external file changes: creating, editing, deleting, or renaming `.yml`/`.md`/`.pdf` files in the opened folder updates the open library live, and renames keep the affected entries (selection and undo history survive). [#738](https://github.com/JabRef/jabref-koppor/pull/738)
 - We added "Open folder as library" (File menu): a folder of PDFs and Hayagriva sidecar files (`.yml`, or `.md` notes with a Hayagriva frontmatter) opens as a library, and it is reopened on the next start. PDFs without a sidecar appear right away and get their metadata extracted in the background. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737)
 - We added `jabkit git merge-driver`, a Git merge driver that merges `.bib` files semantically. [#16838](https://github.com/JabRef/jabref/pull/16838)

docs/requirements/directory-library.md

@@ -34,6 +34,28 @@ the last-opened list and routed back through the directory-library opener.
 
 Needs: impl
 
+## User changes are written back into the sidecar files
+`req~directory-library.write-back~2`
+
+A directory library persists into its sidecar files: user edits rewrite the entry's
+file read-modify-write (content JabRef does not understand survives, including body sections of
+Markdown sidecars under foreign headings), the first user edit of an
+entry without a sidecar creates a Markdown sidecar `X.md` (next to its PDF, sharing the base
+name, or named after the citation key) whose frontmatter carries the Hayagriva data and whose
+markdownlint-clean body carries the comment fields (`# Notes` intro for the comment, one
+`## comment-<name>` section per per-user comment); in plain `.yml` sidecars the comment fields
+are written as extension keys. A citation-key edit renames the YAML map key, and deleting an
+entry removes it
+from its file — the file itself is trashed/deleted once its last entry is gone, the paired PDF
+is never touched. Writes are debounced per file with a trailing-edge debounce that is re-armed
+by every change event — including the keystroke events the CoarseChangeFilter marks as
+filtered, so the tail of a typing burst is never lost; Save (Ctrl+S) flushes them and must never
+write a `.bib` file ("Save as" remains the explicit `.bib` snapshot). Closing needs no save
+prompt. System-initiated changes (background enrichment, generated citation keys, inbound
+synchronization) do not create or rewrite sidecars.
+
+Needs: impl
+
 ## External file changes appear live in an open directory library
 `req~directory-library.inbound-sync~2`
 

jabgui/src/main/java/org/jabref/gui/LibraryTab.java

@@ -63,6 +63,7 @@
 import org.jabref.logic.ai.AiService;
 import org.jabref.logic.citationstyle.CitationStyleCache;
 import org.jabref.logic.command.CommandSelectionTab;
+import org.jabref.logic.directorylibrary.DirectoryLibrarySynchronizer;
 import org.jabref.logic.git.diff.GitDiffChecker;
 import org.jabref.logic.git.util.GitHandlerRegistry;
 import org.jabref.logic.importer.FetcherClientException;
@@ -597,9 +598,7 @@ public void updateTabTitle(boolean isChanged) {
                     tabTitle.append(Localization.lang("untitled"));
                 }
             } else if (databaseLocation == DatabaseLocation.DIRECTORY) {
-                if (isChanged) {
-                    tabTitle.append('*');
-                }
+                // No modification marker: changes are written back to the sidecars continuously
                 bibDatabaseContext.getDirectoryLibraryRoot().ifPresent(root -> {
                     tabTitle.append(root.getFileName().toString());
                     toolTipText.append(root.toAbsolutePath());
@@ -609,8 +608,7 @@ public void updateTabTitle(boolean isChanged) {
                 addSharedDbInformation(toolTipText, bibDatabaseContext);
             }
             addModeInfo(toolTipText, bibDatabaseContext);
-            if ((databaseLocation == DatabaseLocation.LOCAL || databaseLocation == DatabaseLocation.DIRECTORY)
-                    && bibDatabaseContext.getDatabase().hasEntries()) {
+            if ((databaseLocation == DatabaseLocation.LOCAL) && bibDatabaseContext.getDatabase().hasEntries()) {
                 addChangedInformation(toolTipText);
             }
         }
@@ -809,13 +807,21 @@ private boolean showDeleteConfirmationDialog(int numberOfEntries) {
     }
 
     public boolean requestClose() {
-        // DIRECTORY prompts as well: until file write-back exists, edits are in-memory only
-        if (bibDatabaseContext.getLocation() == DatabaseLocation.LOCAL
-                || bibDatabaseContext.getLocation() == DatabaseLocation.DIRECTORY) {
+        if (bibDatabaseContext.getLocation() == DatabaseLocation.LOCAL) {
             if (isModified()) {
                 return confirmClose();
             }
         }
+        if (bibDatabaseContext.getLocation() == DatabaseLocation.DIRECTORY) {
+            // Edits are persisted into the sidecar files; only a failed write needs the user
+            List<Path> unwritable = Optional.ofNullable(bibDatabaseContext.getDirectorySynchronizer())
+                                            .map(DirectoryLibrarySynchronizer::flush)
+                                            .orElse(List.of());
+            return unwritable.isEmpty() || dialogService.showConfirmationDialogAndWait(
+                    Localization.lang("Close library"),
+                    Localization.lang("Could not write the changes to the following files: %0", SaveDatabaseAction.joinPaths(unwritable)),
+                    Localization.lang("Close anyway"));
+        }
         return true;
     }
 

jabgui/src/main/java/org/jabref/gui/exporter/SaveDatabaseAction.java

@@ -32,6 +32,7 @@
 import org.jabref.gui.preferences.GuiPreferences;
 import org.jabref.gui.util.FileDialogConfiguration;
 import org.jabref.gui.util.UiTaskExecutor;
+import org.jabref.logic.directorylibrary.DirectoryLibrarySynchronizer;
 import org.jabref.logic.exporter.AtomicFileWriter;
 import org.jabref.logic.exporter.BibDatabaseWriter;
 import org.jabref.logic.exporter.BibWriter;
@@ -261,7 +262,26 @@ private Optional<Path> askForSavePath() {
         return selectedPath;
     }
 
+    public static String joinPaths(List<Path> files) {
+        return files.stream().map(Path::toString).collect(Collectors.joining("\n"));
+    }
+
     private SaveResult save(BibDatabaseContext bibDatabaseContext, SaveDatabaseMode mode, boolean mayAutoCommit) {
+        if (bibDatabaseContext.getLocation() == DatabaseLocation.DIRECTORY) {
+            // A directory library persists into its sidecar files; saving means flushing the
+            // debounced writes, never writing a .bib ("Save as" remains the explicit snapshot)
+            // [impl->req~directory-library.write-back~2]
+            List<Path> unwritable = Optional.ofNullable(bibDatabaseContext.getDirectorySynchronizer())
+                                            .map(DirectoryLibrarySynchronizer::flush)
+                                            .orElse(List.of());
+            if (!unwritable.isEmpty()) {
+                dialogService.showErrorDialogAndWait(Localization.lang("Save library"),
+                        Localization.lang("Could not write the changes to the following files: %0", joinPaths(unwritable)));
+                return SaveResult.FAILURE;
+            }
+            dialogService.notify(Localization.lang("Library saved"));
+            return SaveResult.SUCCESS;
+        }
         Optional<Path> databasePath = bibDatabaseContext.getDatabasePath();
         if (databasePath.isEmpty()) {
             Optional<Path> savePath = askForSavePath();

jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDirectoryLibraryAction.java

@@ -1,5 +1,7 @@
 package org.jabref.gui.importer.actions;
 
+import java.io.IOException;
+import java.nio.file.Files;
 import java.nio.file.Path;
 
 import javafx.event.Event;
@@ -11,6 +13,7 @@
 import org.jabref.gui.StateManager;
 import org.jabref.gui.actions.SimpleCommand;
 import org.jabref.gui.clipboard.ClipBoardManager;
+import org.jabref.gui.desktop.os.NativeDesktop;
 import org.jabref.gui.preferences.GuiPreferences;
 import org.jabref.gui.util.DirectoryDialogConfiguration;
 import org.jabref.gui.util.UiTaskExecutor;
@@ -109,6 +112,20 @@ private void scanAndShow(Path root) {
                       .executeWith(taskExecutor);
     }
 
+    /// Sidecar files whose last entry was deleted are trashed or deleted per the preference;
+    /// the paired PDF is never touched.
+    private void disposeFile(Path file) {
+        try {
+            if (preferences.getFilePreferences().moveToTrash() && NativeDesktop.get().moveToTrashSupported()) {
+                NativeDesktop.get().moveToTrash(file);
+            } else {
+                Files.delete(file);
+            }
+        } catch (IOException e) {
+            LOGGER.error("Could not remove sidecar {}", file, e);
+        }
+    }
+
     private void showLibraryTab(DirectoryLibraryScanner.ScanResult scanResult, PdfEntryFactory pdfEntryFactory) {
         // The synchronous factory keeps the DIRECTORY location: the ParserResult-based one
         // reconstructs a fresh (LOCAL) context from database + metadata on loading success
@@ -130,7 +147,8 @@ private void showLibraryTab(DirectoryLibraryScanner.ScanResult scanResult, PdfEn
 
         BibDatabaseContext databaseContext = scanResult.databaseContext();
         DirectoryLibrarySynchronizer synchronizer = new DirectoryLibrarySynchronizer(
-                databaseContext, scanResult.catalog(), pdfEntryFactory, UiTaskExecutor::runInJavaFXThread);
+                databaseContext, scanResult.catalog(), pdfEntryFactory, this::disposeFile,
+                UiTaskExecutor::runInJavaFXThread);
         databaseContext.attachDirectorySynchronizer(synchronizer);
         synchronizer.startWatching(Injector.instantiateModelOrService(DirectoryMonitor.class));
 

jablib/src/main/java/org/jabref/logic/directorylibrary/DirectoryLibraryCatalog.java

@@ -6,6 +6,7 @@
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
+import java.util.Set;
 
 import org.jabref.model.entry.BibEntry;
 
@@ -36,6 +37,23 @@ public Optional<EntrySource> sourceOf(BibEntry entry) {
         return Optional.ofNullable(sourceByEntryId.get(entry.getId()));
     }
 
+    public void removeEntry(String entryId) {
+        Optional.ofNullable(sourceByEntryId.remove(entryId)).ifPresent(source ->
+                entryIdsByFile.computeIfPresent(source.yamlFile(), (_, ids) -> {
+                    ids.remove(entryId);
+                    return ids.isEmpty() ? null : ids;
+                }));
+    }
+
+    /// Records the Hayagriva key the entry was last written under (after a citation-key edit).
+    public void updateHayagrivaKey(BibEntry entry, String hayagrivaKey) {
+        sourceByEntryId.computeIfPresent(entry.getId(), (_, source) -> new EntrySource(source.yamlFile(), hayagrivaKey));
+    }
+
+    public Set<Path> files() {
+        return Set.copyOf(entryIdsByFile.keySet());
+    }
+
     /// Entry ids of all entries read from the given file, in file order.
     public List<String> entryIdsIn(Path yamlFile) {
         return List.copyOf(entryIdsByFile.getOrDefault(yamlFile, List.of()));