Feature: Lazy Loading Architecture for Large Repository Performance

CHANGELOG.md

@@ -1,3 +1,54 @@
+## [1.2.0] - 2025-01-10
+
+### Added
+
+- **Lazy Loading Architecture for Improved Performance:**
+  - Implemented lightweight metadata-only initial file scanning that defers expensive file I/O and tokenization
+  - Added token estimation function that calculates approximate token counts based on file extension and size without reading file content
+  - Files now display estimated token counts with a visual indicator (~tokens and "est" badge) until actual tokenization occurs
+  - Individual files are processed for real tokens only when selected, dramatically improving initial load times
+
+- **Visual Processing Indicators:**
+  - Added loading spinner next to individual files being processed for token counting
+  - Implemented full-page processing overlay for batch operations (reserved for future use)
+  - Clear visual distinction between estimated and actual token counts in both TreeItem and FileCard components
+
+### Improved
+
+- **Large Repository Handling:**
+  - All folders now use lightweight scanning by default, making the app responsive even with massive codebases
+  - Large folder warning now triggers instantly based on file count estimates rather than after full processing
+  - Token threshold warning uses more realistic estimates (~2000 tokens per file instead of 100)
+  - Removed automatic batch processing from folder selection to maintain instant UI response
+
+- **Performance Optimizations:**
+  - Optimized file path comparison using Set-based lookups (O(1)) instead of array iterations (O(n²))
+  - Deferred content reading and tokenization until files are actually needed
+  - Significantly reduced memory usage by not loading file contents until selected
+
+### Fixed
+
+- **UI Responsiveness:**
+  - Fixed issue where selecting folders would freeze the UI while processing all files
+  - Fixed double-click requirement for first-time folder expansion in tree view
+  - Token counts now properly update when files are selected and processed
+
+## [1.1.2] - 2025-06-18
+
+### Added
+
+- **Large Folder Warning System:**
+  - Added a warning modal that appears when selecting folders with more than 500,000 tokens, alerting users about potential performance impacts.
+  - Users can choose to proceed with full selection, load files but keep them deselected, or cancel the folder load operation.
+  - Prevents application freezing when loading extremely large repositories.
+
+### Improved
+
+- **Default Folder Tree State:**
+  - Folder tree now defaults to collapsed state instead of expanded, improving initial load experience for large projects.
+  - Only the root folder is expanded by default when a new folder is selected.
+  - This change significantly improves the user experience when working with projects containing many nested folders.
+
 ## [1.1.1] - 2025-05-23
 
 ### Added

CONTRIBUTING.md

@@ -1,36 +1,76 @@
 # Contributing to PasteMax
 
-Thank you for considering contributing to PasteMax! This document outlines the process for contributing to the project.
+Thank you for considering contributing to PasteMax! We welcome all contributions, from bug reports to feature requests and code changes.
 
 ## Getting Started
 
-1. Fork the repository
-2. Clone your fork: `git clone https://github.com/yourusername/pastemax.git`
-3. Create a branch for your changes: `git checkout -b feature/your-feature-name`
-4. Install dependencies: `npm install`
-5. Make your changes
-6. Run tests and linting: `npm run lint`
-7. Commit your changes with a descriptive message
+1.  **Fork the repository** on GitHub.
+2.  **Clone your fork** locally:
+    ```bash
+    git clone https://github.com/your-username/pastemax.git
+    cd pastemax
+    ```
+3.  **Install dependencies**:
+    ```bash
+    npm install
+    ```
+4.  **Create a new branch** for your feature or bugfix:
+    ```bash
+    git checkout -b feature/your-amazing-feature
+    ```
 
-## Pull Request Process
+## Development Workflow
 
-1. Update the README.md or documentation with details of your changes if appropriate
-2. Make sure your code passes all linting checks
-3. Submit a pull request to the main repository
-4. The maintainers will review your PR as soon as possible
+To run the application in development mode with hot-reloading for both the frontend (React) and backend (Electron), use the following command:
 
-## Code Style
+```bash
+npm run dev:all
+```
 
-- Follow the existing code style in the project
-- Run `npm run lint` to ensure your code meets the project's style guidelines
-- Comment your code where appropriate
+This will:
+- Start the Vite dev server for the React app.
+- Start the Electron main process, which will load the app from the Vite server.
+- Automatically restart the Electron process when you make changes to files in the `electron/` directory.
 
-## Issues
+### Scripts
 
-- Use GitHub Issues to report bugs or suggest new features
-- Check existing issues before creating a new one
-- When reporting bugs, include steps to reproduce, expected behavior, and actual behavior
+- `npm run dev`: Starts only the Vite dev server.
+- `npm run dev:electron`: Starts only the Electron process (expects the Vite server to be running).
+- `npm run lint`: Runs ESLint to check for code quality issues.
+- `npm run lint:fix`: Attempts to automatically fix linting issues.
+- `npm run format:all`: Formats all code using Prettier.
+
+## Making Changes
+
+1.  Make your code changes in your feature branch.
+2.  Ensure your code follows the existing style and conventions.
+3.  Add or update documentation in the `docs/` directory if you are changing functionality.
+4.  Run the linter to ensure your code is clean:
+    ```bash
+    npm run lint
+    ```
+5.  Commit your changes with a clear and descriptive commit message.
+
+## Submitting a Pull Request
+
+1.  Push your feature branch to your fork on GitHub:
+    ```bash
+    git push origin feature/your-amazing-feature
+    ```
+2.  Open a **Pull Request** from your feature branch to the `main` branch of the `kleneway/pastemax` repository.
+3.  Provide a clear title and description for your pull request, explaining the changes you've made and why.
+4.  The maintainers will review your PR as soon as possible. Thank you for your contribution!
+
+## Reporting Issues
+
+- Use the [GitHub Issues](https://github.com/kleneway/pastemax/issues) page to report bugs or suggest new features.
+- Before creating a new issue, please check if a similar one already exists.
+- When reporting bugs, please include:
+  - Steps to reproduce the issue.
+  - The expected behavior.
+  - The actual behavior.
+  - Your operating system and app version.
 
 ## License
 
-By contributing to PasteMax, you agree that your contributions will be licensed under the project's MIT License.
+By contributing to PasteMax, you agree that your contributions will be licensed under the project's [MIT License](LICENSE).

README.md

@@ -82,54 +82,39 @@ npm install
 
 ```
 npm run build:electron
-npm run package
 ```
 
-**Note**: If you encounter issues with `npm run package`, you can try the platform-specific command:
-
+4. Package the app for your platform:
 ```
-npm run package:win
+# For macOS
 npm run package:mac
-npm run package:linux
-```
-
-After successful build, you'll find the executable files inside the `release-builds` directory:
-
-**Windows:**
-
-- `PasteMax Setup 1.0.0.exe` - Installer version
-- `PasteMax 1.0.0.exe` - Portable version
-
-**Mac:**
 
-- `PasteMax 1.0.0.dmg` - Installer version
-- `PasteMax 1.0.0.zip` - Portable version
+# For Windows
+npm run package:win
 
-**Linx:**
+# For Linux
+npm run package:linux
+```
 
-- `PasteMax 1.0.0.deb` - Installer version (Deb package)
-- `PasteMax 1.0.0.rpm` - Installer version (RPM package)
-- `PasteMax 1.0.0.AppImage` - Portable version
+After successful packaging, you'll find the executable files inside the `release-builds` directory.
 
 ## Development
 
 ### Prerequisites
 
-- Node.js (v14 or higher)
-- npm or yarn
+- Node.js (v18 or higher)
+- npm
 
 ### Running in Development Mode
 
-To run the application in development mode:
+To run the application in development mode with hot-reloading for both the frontend and backend:
 
 ```
-# Start the Vite dev server
-npm run dev
-
-# In a separate terminal, start Electron
-npm run dev:electron
+npm run dev:all
 ```
 
+This will start the Vite dev server for the React app and the Electron main process concurrently.
+
 ### Building for Production
 
 To build the application for production:
@@ -139,60 +124,51 @@ To build the application for production:
 npm run build:electron
 
 # Create platform-specific distributables
-npm run package
+npm run package:mac    # macOS
+npm run package:win    # Windows  
+npm run package:linux  # Linux
 ```
 
 ## Project Structure
 
-- `src/` - React application source code
-  - `components/` - React components
-  - `context/` - React context providers
-  - `hooks/` - Custom React hooks
+- **`src/`** - React application source code (Renderer Process)
+  - `components/` - Reusable React components
+  - `context/` - React context providers (e.g., ThemeContext)
+  - `hooks/` - Custom React hooks for stateful logic
   - `types/` - TypeScript type definitions
-  - `utils/` - Utility functions
-  - `styles/` - CSS styles
-  - `assets/` - Static assets like images
-- `electron/` - Electron-Backend related files
-  - `main.js` - Electron main process
-  - `preload.js` - Preload script for secure IPC
-  - `renderer.js` - Renderer process utilities
-  - `build.js` - Build script for production
-  - `dev.js` - Development script
-  - `excluded-files.js` - Configuration for files to exclude by default
-  - `file-processor.js` - File processing utilities
-  - `ignore-manager.js` - Ignore pattern management
-  - `update-checker.js` - Update checking functionality
-  - `update-manager.js` - Update management
-  - `utils.js` - Utility functions
-  - `watcher.js` - File change watcher
-- `public/` - Public assets (favicon, etc.)
-- `scripts/` - Utility scripts for building and testing
-- `docs/` - Documentation
+  - `utils/` - Utility functions for the frontend
+  - `styles/` - Modularized CSS stylesheets
+- **`electron/`** - Electron-Backend related files (Main Process)
+  - `main.js` - Main process entry point, window management, and IPC handling
+  - `preload.js` - Preload script for secure IPC communication
+  - `file-processor.js` - Logic for reading files, counting tokens, etc.
+  - `ignore-manager.js` - Logic for handling `.gitignore` and other ignore patterns
+  - `update-manager.js` - Logic for managing update checks
+  - `watcher.js` - File system watcher logic using Chokidar
+- **`public/`** - Static assets (e.g., icons)
+- **`scripts/`** - Utility scripts for building, testing, and debugging
 
 ## Libraries Used
 
-- Electron - Desktop application framework
-- React - UI library
-- TypeScript - Type safety
-- Vite - Build tool and development server
-- tiktoken - Token counting for LLM context estimation
-- ignore - .gitignore-style pattern matching for file exclusions
-- chokidar - File Watcher
+- **Electron** - Desktop application framework
+- **React** & **TypeScript** - For building the user interface
+- **Vite** - Build tool and development server
+- **Tiktoken** - Fast BPE tokenization for LLM context estimation
+- **ignore** - For `.gitignore`-style pattern matching
+- **Chokidar** - Advanced file system watcher
 
 ## Troubleshooting
 
 ### Getting "Warning: Not trusted" on Windows
 
-If you see a warning about the app not being trusted, you can bypass this by clicking "run anyways". This is a common issue with Electron apps, especially since PasteMax is not signed.
-
-### Getting "App not responding" on Mac
+If you see a warning about the app not being trusted, you can bypass this by clicking "More info" -> "Run anyway". This is a common issue with unsigned Electron apps.
 
-If you encounter an "App not responding" message on Mac, it may be due to macOS security settings. You can try the following:
+### Getting "App can't be opened" on Mac
 
-1. Open System Preferences.
-2. Go to Security & Privacy.
-3. Under the General tab, look for the "Allow apps downloaded from" section.
-4. Look for "PasteMax" and click "Open Anyway".
+If you encounter this message on macOS, it may be due to security settings.
+1. Right-click the `PasteMax.app` file and select "Open".
+2. You may see a warning dialog. Click "Open" again to confirm.
+You should only need to do this the first time you run the app.
 
 ### Other Issues
 
@@ -204,13 +180,7 @@ MIT License - see the [LICENSE](LICENSE) file for details.
 
 ## Contributing
 
-Contributions are welcome! Please feel free to submit a Pull Request.
-
-1. Fork the repository
-2. Create your feature branch (`git checkout -b feature/amazing-feature`)
-3. Commit your changes (`git commit -m 'Add some amazing feature'`)
-4. Push to the branch (`git push origin feature/amazing-feature`)
-5. Open a Pull Request
+Contributions are welcome! Please see the [CONTRIBUTING.md](CONTRIBUTING.md) file for details.
 
 ## Star History ⭐
 

TODO.md

@@ -24,4 +24,4 @@
 - [ ] Explore alternative libraries for file tree navigation
 - [ ] Consider using a different library for the file tree navigation
 - [ ] Chokidar for file watching
-- [ ] Zustand for state management
+- [ ] Zustand for state management