Chore/ux improvement bug fixes

MEMORY.md

@@ -34,6 +34,7 @@ Architecture: Modular Hexagonal/DDD.
 * Security: Remediated Medium severity security findings (secured Swagger UI in production, sanitized global exception handler, downgraded JWT logging, and unified login error messages).
 * System: Configured Docker containers (backend, frontend, db) to automatically restart after server reboot using 'restart: unless-stopped' policy in docker-compose.yml.
 * Backend: Implemented dynamic Image Consensus Threshold feature using an asynchronous `ThresholdEventListener` that intercepts `ClassificationSubmittedEvent`. It evaluates the weighted credibility of classifications and automatically excludes images from distribution (via `ConsensusResult` tracking) once the dynamic researcher-defined threshold is met.
+* Backend/Frontend: Optimized the CSV Export System (Issue #257). Implemented a non-blocking, background-execution export flow in the frontend using a global Zustand store and an animated `GlobalDownloadToast`. Cleaned up the backend CSV generation (`ExportService`) to significantly reduce file size and improve traceability by removing the raw base64 image strings and replacing them with explicit local and external IDs (`swipelab_image_id`, `stardbi_experiment_id`, `stardbi_image_id`, `stardbi_crop_id`).
 ## Current Focus (Active GitHub Issues)
 * Issue #201: Refactor Backend roles to include Researchers and Super Admin.
 * Issue #226: [Frontend] fix recipients list not deleting users.

backend/src/docs/analytics.md

@@ -28,3 +28,8 @@ A heavily cached, expensive query that provides a comprehensive platform snapsho
 - **Time Windows**: Shows activity (classifications, unique users, tasks) for "Today", "This Week", and "This Month".
 - **Confidence Trend**: Tracks average user credibility scores over a 30-day window.
 - **Label Distribution**: Shows the split of YES / NO / UNSURE labels over time.
+
+## 6. Data Export (CSV & JSON)
+Researchers and SuperAdmins can export classification datasets in CSV and JSON formats. These exports are highly optimized for both performance and readability:
+- **Traceability**: All exported rows include strict cross-system mapping identifiers (`swipelab_image_id`, `stardbi_experiment_id`, `stardbi_image_id`, and `stardbi_crop_id`).
+- **Optimization**: Raw Base64 image strings are completely omitted from the generated files, vastly reducing network latency and file sizes (e.g., bringing a row payload down from ~100+ KB to ~150 bytes).

backend/src/docs/api/analytics-api.md

@@ -25,3 +25,12 @@ Base path: `/api/v1/admin/export`
 | `GET` | `/tasks/{taskId}/summary` | Export task summary |
 | `GET` | `/tasks/{taskId}/csv` | Export task data as CSV |
 | `GET` | `/tasks/{taskId}/json` | Export task data as JSON |
+
+### Export Data Schema
+
+The CSV and JSON exports are optimized for performance and file size by omitting raw Base64 image strings. They contain the following cross-system traceability identifiers to reliably map classifications back to their source:
+
+- `swipelab_image_id`: Internal primary key of the image in the SwipeLab database.
+- `stardbi_experiment_id`: External ID mapping to the StarDBi experiment.
+- `stardbi_image_id`: External ID mapping to the StarDBi parent image.
+- `stardbi_crop_id`: External ID mapping to the StarDBi bounding box/crop.

backend/src/docs/metrics.md

@@ -0,0 +1,58 @@
+# SwipeLab Analytics Metrics Documentation
+
+This document explains the mathematical formulas and backend implementation logic behind the analytics data generated by `AnalyticsService.java` for the Researcher Dashboard.
+
+## 1. User Accuracy (Gold Accuracy)
+
+### What it is
+The accuracy score tracks how effectively a user is identifying correct classifications, serving as a primary indicator of data labeling quality. 
+
+### How it is calculated
+Since normal images in the dataset do not have a known "correct" answer, a user's accuracy is **only evaluated based on their performance against Gold Images** (test images with pre-verified answers).
+
+* **Formula**: `Correct Gold Classifications / Total Gold Classifications`
+* **Backend Source**: `ClassificationFactRepository.java`
+* **Implementation Note**: The system specifically filters for `c.isCorrect IS NOT NULL` to determine `Total Gold Classifications`. Normal classifications (where `isCorrect` is null) are completely excluded from the denominator to prevent artificially deflating the user's accuracy score to ~0.0%.
+
+## 2. Participation Metrics
+
+### What it is
+These metrics measure engagement volume and speed for a specific classification task.
+
+### How it is calculated
+* **Active Users**: The count of distinct, unique `userId` values associated with the `ClassificationFact` entries for the task.
+* **Total Classifications**: The absolute total count of all `ClassificationFact` rows (swipes) executed against this task.
+* **Average Classifications per User**: `Total Classifications / Active Users`.
+* **Median Response Time**: 
+    * The backend retrieves the `responseTimeMs` (the time taken between an image appearing on screen and the user swiping) for every single classification fact.
+    * Null values are filtered out.
+    * The list is sorted, and the median value (middle element, or average of two middle elements) is returned.
+
+## 3. Data Quality Metrics
+
+### What it is
+These metrics evaluate the overall trustworthiness of the labels being generated for a task based on user credibility.
+
+### How it is calculated
+* **Average Credibility**: 
+    * Every time a user makes a classification, their *current credibility score at that exact moment* is snapshotted into `credibilityAtTime`.
+    * The backend calculates the global average of `credibilityAtTime` across all facts in the task.
+* **Low Quality Users**:
+    * The system groups all classification facts by `userId` and calculates the average credibility for each specific user within this task.
+    * A user is flagged as a "Low Quality User" if their average credibility score drops below the threshold of **50.0/100**.
+    * The final metric returns the absolute count of users who fall below this threshold.
+
+## 4. Consensus Metrics
+
+### What it is
+Consensus evaluates how strongly the crowd agrees on the labels for the images in the task.
+
+### How it is calculated
+* **Overall Average Consensus**:
+    * The backend groups all `ClassificationFact` rows by `imageId`.
+    * For each image, it extracts the `consensusScore` (taking the maximum/latest recorded score for that image).
+    * It then averages these scores across all classified images in the task.
+    * The result is represented as a percentage (0-100%).
+* **Low Consensus Images**:
+    * While calculating the above, the system checks each image's `consensusScore` against a strict threshold (currently `0.8` or 80%).
+    * Any image with a score below this threshold increments the "Low Consensus Images" counter. This highlights complex, ambiguous, or highly-debated images that may require researcher intervention.

backend/src/main/java/com/swipelab/analytics/application/AnalyticsEventListener.java

@@ -76,10 +76,14 @@ private void updateUserDailyStats(ClassificationSubmittedEvent event, LocalDate
                         .build());
 
         stats.setTotal(stats.getTotal() + 1);
-        if (event.isCorrect()) {
+        if (event.isGoldStandard() && event.isCorrect()) {
             stats.setCorrect(stats.getCorrect() + 1);
         }
-        stats.setAccuracy(stats.getTotal() > 0 ? (double) stats.getCorrect() / stats.getTotal() : 0.0);
+
+        // Note: For true daily accuracy we need a count of gold classifications for the day.
+        // For now, we will leave the denormalized accuracy at 0.0 or calculate a rough estimate,
+        // since the UI fetches live accuracy from the facts table directly.
+        stats.setAccuracy(0.0);
 
         userDailyStatsRepository.save(stats);
     }

backend/src/main/java/com/swipelab/analytics/application/AnalyticsService.java

@@ -5,6 +5,7 @@
 import com.swipelab.analytics.infrastructure.*;
 import com.swipelab.classification.domain.core.Classification.UserResponse;
 import com.swipelab.classification.infrastructure.ClassificationRepository;
+import com.swipelab.classification.infrastructure.ConsensusResultRepository;
 import com.swipelab.classification.infrastructure.ImageRepository;
 import com.swipelab.config.CacheConfig;
 import com.swipelab.analytics.dto.DashboardStatsResponse;
@@ -43,6 +44,7 @@ public class AnalyticsService {
     private final UserRepository userRepository;
     private final TaskRepository taskRepository;
     private final ImageRepository imageRepository;
+    private final ConsensusResultRepository consensusResultRepository;
 
     // ─── User-scoped endpoints ────────────────────────────────────────────────
 
@@ -62,7 +64,8 @@ public UserProgressResponse getUserProgress(String userId) {
             }
         }
 
-        double accuracy = total > 0 ? (double) correct / total : 0.0;
+        Double calculatedAccuracy = classificationFactRepository.getUserAccuracy(userId);
+        double accuracy = calculatedAccuracy != null ? calculatedAccuracy : 0.0;
 
         return UserProgressResponse.builder()
                 .completed((int) total)
@@ -83,7 +86,9 @@ public UserStatisticsResponse getUserStatistics(String userId) {
                 correct = row[1] != null ? ((Number) row[1]).longValue() : 0;
             }
         }
-        double accuracy = total > 0 ? (double) correct / total : 0.0;
+
+        Double calculatedAccuracy = classificationFactRepository.getUserAccuracy(userId);
+        double accuracy = calculatedAccuracy != null ? calculatedAccuracy : 0.0;
 
         UserRanking ranking = userRankingRepository.findByUserIdAndPeriod(userId, "ALL_TIME")
                 .orElse(UserRanking.builder().rank(0).percentile(0).build());
@@ -194,7 +199,8 @@ public TimeSeriesResponse getTimeSeries(String userId, String metric, String per
 
     @Transactional(readOnly = true)
     public TaskAnalyticsResponse getTaskAnalytics(Long taskId) {
-        Long completedImages = classificationFactRepository.countCompletedImages(taskId);
+        Long imagesClassified = classificationFactRepository.countDistinctImagesByTaskId(taskId);
+        Long completedImages = consensusResultRepository.countCompletedImagesByTaskId(taskId);
         List<ClassificationFact> facts = classificationFactRepository.findByTaskId(taskId);
         int totalClassifications = facts.size();
 
@@ -204,7 +210,7 @@ public TaskAnalyticsResponse getTaskAnalytics(Long taskId) {
                 : 0.0;
 
         TaskAnalyticsResponse.Progress progress = TaskAnalyticsResponse.Progress.builder()
-                .imagesClassified(completedImages.intValue())
+                .imagesClassified(imagesClassified.intValue())
                 .totalImages(totalImages)
                 .completedImages(completedImages.intValue())
                 .percentComplete(percentComplete)
@@ -225,15 +231,88 @@ public TaskAnalyticsResponse getTaskAnalytics(Long taskId) {
                         .build())
                 .collect(Collectors.toList());
 
+        // ─── Participation ──────────────────
+        long activeUsers = facts.stream().map(ClassificationFact::getUserId).distinct().count();
+        int avgClassifications = activeUsers > 0 ? (int) (totalClassifications / activeUsers) : 0;
+
+        List<Long> responseTimes = facts.stream()
+                .map(ClassificationFact::getResponseTimeMs)
+                .filter(java.util.Objects::nonNull)
+                .sorted()
+                .collect(Collectors.toList());
+        long medianResponseTime = 0L;
+        if (!responseTimes.isEmpty()) {
+            int mid = responseTimes.size() / 2;
+            medianResponseTime = responseTimes.size() % 2 == 1 
+                ? responseTimes.get(mid) 
+                : (responseTimes.get(mid - 1) + responseTimes.get(mid)) / 2;
+        }
+
+        TaskAnalyticsResponse.Participation participation = TaskAnalyticsResponse.Participation.builder()
+                .activeUsers((int) activeUsers)
+                .totalClassifications(totalClassifications)
+                .averageClassificationsPerUser(avgClassifications)
+                .medianResponseTimeMs(medianResponseTime)
+                .build();
+
+        // ─── Quality ────────────────────────
+        double sumCredibility = 0;
+        int credibilityCount = 0;
+        Map<String, List<Double>> userCredibility = new HashMap<>();
+        for (ClassificationFact f : facts) {
+            if (f.getCredibilityAtTime() != null) {
+                sumCredibility += f.getCredibilityAtTime();
+                credibilityCount++;
+                userCredibility.computeIfAbsent(f.getUserId(), k -> new ArrayList<>()).add(f.getCredibilityAtTime());
+            }
+        }
+        double avgCredibility = credibilityCount > 0 ? sumCredibility / credibilityCount : 0.0;
+
+        int lowQualityCount = 0;
+        for (List<Double> creds : userCredibility.values()) {
+            double uAvg = creds.stream().mapToDouble(Double::doubleValue).average().orElse(100.0);
+            if (uAvg < 50.0) {
+                lowQualityCount++;
+            }
+        }
+
+        TaskAnalyticsResponse.Quality quality = TaskAnalyticsResponse.Quality.builder()
+                .averageCredibility(avgCredibility)
+                .lowQualityUsers(lowQualityCount)
+                .build();
+
+        // ─── Consensus ──────────────────────
+        Map<Long, Double> imageConsensus = new HashMap<>();
+        for (ClassificationFact f : facts) {
+            if (f.getConsensusScore() != null) {
+                imageConsensus.merge(f.getImageId(), f.getConsensusScore(), Math::max);
+            }
+        }
+        double sumConsensus = 0;
+        int lowConsensusImages = 0;
+        for (Double score : imageConsensus.values()) {
+            sumConsensus += score;
+            if (score < 0.8) {
+                lowConsensusImages++;
+            }
+        }
+        double avgConsensus = imageConsensus.isEmpty() ? 0.0 : (sumConsensus / imageConsensus.size()) * 100.0;
+
+        TaskAnalyticsResponse.Consensus consensus = TaskAnalyticsResponse.Consensus.builder()
+                .overallAverage(avgConsensus)
+                .lowConsensusImages(lowConsensusImages)
+                .threshold(0.8)
+                .build();
+
         return TaskAnalyticsResponse.builder()
                 .taskId(taskId)
                 .status("ACTIVE")
                 .progress(progress)
                 .speciesAnalytics(saList)
                 .generatedAt(java.time.LocalDateTime.now().toString())
-                .consensus(TaskAnalyticsResponse.Consensus.builder().build())
-                .participation(TaskAnalyticsResponse.Participation.builder().build())
-                .quality(TaskAnalyticsResponse.Quality.builder().build())
+                .consensus(consensus)
+                .participation(participation)
+                .quality(quality)
                 .timeSeries(List.of())
                 .build();
     }
@@ -355,14 +434,15 @@ private UserPerformanceResponse mapToUserPerformanceResponse(Object[] row) {
         String username = (String) row[0];
         int total = row[1] != null ? ((Number) row[1]).intValue() : 0;
         int correct = row[2] != null ? ((Number) row[2]).intValue() : 0;
-        double avgCredibility = row[3] != null ? ((Number) row[3]).doubleValue() : 0.0;
-        double accuracy = total > 0 ? (double) correct / total : 0.0;
+        int goldTotal = row[3] != null ? ((Number) row[3]).intValue() : 0;
+        double avgCredibility = row[4] != null ? ((Number) row[4]).doubleValue() : 0.0;
+        double accuracy = goldTotal > 0 ? (double) correct / goldTotal : 0.0;
 
         return UserPerformanceResponse.builder()
                 .username(username)
                 .displayName(username)
                 .totalClassifications(total)
-                .goldImageClassifications(0)
+                .goldImageClassifications(goldTotal)
                 .correctGoldClassifications(correct)
                 .goldAccuracy(accuracy)
                 .credibilityScore(avgCredibility)