Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/build-and-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ on:
types: [published]
workflow_dispatch: # Allow manual trigger

permissions:
contents: read
actions: read

jobs:
build:
name: Build Package
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ on:
pull_request:
branches: [main, dev]

permissions:
contents: read
actions: read

jobs:
test:
name: Test (Python ${{ matrix.python-version }} on ${{ matrix.os }})
Expand Down
89 changes: 39 additions & 50 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,16 +65,9 @@ This ensures:

## Installation and Initial Setup

### 1. Install Dependencies
### 1. Install the module
```bash
# Create a virtual environment (recommended)
python -m venv .venv
.venv\Scripts\activate # Windows
# or
source .venv/bin/activate # Linux/macOS

# Install dependencies
pip install -r requirements.txt
pip install email-processor
```

### 2. Create Configuration
Expand All @@ -88,7 +81,9 @@ python -m email_processor config init
### 3. Set Password
```bash
# Set IMAP password (will be prompted interactively)
# --user can be omitted if imap.user is set in config.yaml
python -m email_processor password set --user your_email@example.com
python -m email_processor password set # uses imap.user from config

# Or from file
python -m email_processor password set --user your_email@example.com --password-file ~/.pass --delete-after-read
Expand All @@ -103,15 +98,36 @@ python -m email_processor config validate
python -m email_processor status
```

### 5. First Run
### 5. Fetch (download emails and attachments)
Uses config by default (IMAP server, folder, processing options).
```bash
# Run email processing (test mode without real actions)
# Test mode (no real actions)
python -m email_processor fetch --dry-run

# Real run
# Run fetch
python -m email_processor fetch
```

### 6. Send (email files)
```bash
# Send a single file
python -m email_processor send file /path/to/file.pdf --to recipient@example.com
```

### 7. Send All Files from Folder
Uses config by default (`smtp.send_folder`, `smtp.default_recipient`).
```bash
# Send from folder (config defaults)
python -m email_processor send
# Or explicitly:
python -m email_processor send folder
```

### 8. Full pipeline: fetch + send
```bash
python -m email_processor run
```

---

# 🎯 Usage
Expand All @@ -130,6 +146,7 @@ python -m email_processor run --since 7d --max-emails 100
```

#### Email Fetching Only (without sending)
Uses config (IMAP, processing) by default.
```bash
# Fetch emails and attachments
python -m email_processor fetch
Expand Down Expand Up @@ -169,7 +186,12 @@ python -m email_processor send file file.pdf --to user@example.com --dry-run

#### Send All Files from Folder
```bash
# Send all new files from folder
# With config defaults (smtp.send_folder, smtp.default_recipient)
python -m email_processor send
# Or explicitly:
python -m email_processor send folder

# Explicit path and recipient
python -m email_processor send folder /path/to/folder --to recipient@example.com

# With custom subject
Expand All @@ -185,16 +207,19 @@ python -m email_processor send folder /path/to/folder --to user@example.com --su
#### Set Password
```bash
# Interactive password input
# --user is optional when imap.user is in config.yaml
python -m email_processor password set --user your_email@example.com
python -m email_processor password set # uses imap.user from config

# From file (file will be deleted after reading)
python -m email_processor password set --user your_email@example.com --password-file ~/.pass --delete-after-read
```

#### Clear Password
```bash
# Delete saved password
# Delete saved password (--user optional if imap.user in config)
python -m email_processor password clear --user your_email@example.com
python -m email_processor password clear # uses imap.user from config
```

### Configuration Management
Expand Down Expand Up @@ -744,42 +769,6 @@ Dictionary of regex patterns to folder paths. Emails matching a pattern will be

---

# 🛠️ Features & Improvements

## v7.1 Features
- ✅ **Modular architecture** - Clean separation of concerns
- ✅ **YAML configuration** - Easy configuration management
- ✅ **Keyring password storage** - Secure credential management
- ✅ **Per-day UID storage** - Optimized performance
- ✅ **Two-phase IMAP fetch** - Efficient email processing
- ✅ **Password management commands** - `password set` and `password clear` subcommands
- ✅ **Configuration validation** - Validates config on startup
- ✅ **Structured logging** - JSON and console formats with file output
- ✅ **Configurable logging levels** - DEBUG, INFO, WARNING, ERROR, CRITICAL
- ✅ **Enhanced error handling** - Comprehensive error recovery
- ✅ **Detailed processing statistics** - File type statistics
- ✅ **Progress bar** - Visual progress indicator (tqdm)
- ✅ **File extension filtering** - Whitelist/blacklist support
- ✅ **Disk space checking** - Prevents out-of-space errors
- ✅ **Dry-run mode** - Test without downloading (`--dry-run`)
- ✅ **Type hints** - Full type annotation support
- ✅ **Path traversal protection** - Security hardening
- ✅ **Attachment size validation** - Prevents oversized downloads

---

# 📝 Notes

- The script is **idempotent**: safe to run multiple times
- Processed UIDs are stored per day for optimal performance
- Passwords are securely stored in system keyring
- Configuration is validated on startup
- All errors are logged with appropriate detail levels
- Progress bar shows real-time statistics (processed, skipped, errors)
- File extension filtering helps prevent unwanted downloads
- Disk space is checked before each download (with 10MB buffer)
- Logs are automatically rotated daily when file logging is enabled

# 🏗️ Architecture

The project uses a modular architecture for better maintainability:
Expand Down
42 changes: 28 additions & 14 deletions email_processor/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -159,11 +159,12 @@ def main() -> int:

# Command: password set
if args.command == "password" and args.password_command == "set":
if not args.user:
ui.error("--user is required")
user = args.user or cfg.get("imap", {}).get("user")
if not user:
ui.error("--user is required or set imap.user in config")
return ExitCode.VALIDATION_FAILED
return passwords.set_password(
args.user,
user,
args.password_file if hasattr(args, "password_file") else None,
args.delete_after_read if hasattr(args, "delete_after_read") else False,
config_path,
Expand All @@ -172,10 +173,19 @@ def main() -> int:

# Command: password clear
if args.command == "password" and args.password_command == "clear":
if not args.user:
ui.error("--user is required")
user = args.user or cfg.get("imap", {}).get("user")
if not user:
ui.error("--user is required or set imap.user in config")
return ExitCode.VALIDATION_FAILED
return passwords.clear_passwords(args.user, ui)
return passwords.clear_passwords(user, ui)

# Command: send (default: send folder when no subcommand)
if args.command == "send" and args.send_command is None:
args.send_command = "folder"
if not hasattr(args, "dir"):
args.dir = None
if not hasattr(args, "to"):
args.to = None

# Command: send file
if args.command == "send" and args.send_command == "file":
Expand Down Expand Up @@ -209,16 +219,20 @@ def main() -> int:

# Command: send folder
if args.command == "send" and args.send_command == "folder":
if not hasattr(args, "dir") or not args.dir:
ui.error("Directory path is required")
folder = args.dir if hasattr(args, "dir") else None
to_addr = args.to if hasattr(args, "to") else None
folder = folder or cfg.get("smtp", {}).get("send_folder")
to_addr = to_addr or cfg.get("smtp", {}).get("default_recipient")
if not folder:
ui.error("Directory path is required or set smtp.send_folder in config")
return ExitCode.VALIDATION_FAILED
if not hasattr(args, "to") or not args.to:
ui.error("--to is required")
if not to_addr:
ui.error("--to is required or set smtp.default_recipient in config")
return ExitCode.VALIDATION_FAILED

# Validate email addresses
if not _validate_email(args.to):
ui.error(f"Invalid email address: {args.to}")
if not _validate_email(to_addr):
ui.error(f"Invalid email address: {to_addr}")
return ExitCode.VALIDATION_FAILED
if hasattr(args, "cc") and args.cc and not _validate_email(args.cc):
ui.error(f"Invalid CC email address: {args.cc}")
Expand All @@ -229,8 +243,8 @@ def main() -> int:

return smtp.send_folder(
cfg,
args.dir,
args.to,
folder,
to_addr,
args.subject if hasattr(args, "subject") else None,
args.dry_run,
config_path,
Expand Down
16 changes: 9 additions & 7 deletions email_processor/cli/args.py
Original file line number Diff line number Diff line change
Expand Up @@ -183,13 +183,15 @@ def parse_arguments() -> argparse.Namespace:
send_folder_parser.add_argument(
"dir",
type=str,
help="Directory path containing files to send",
nargs="?",
default=None,
help="Directory path (default: smtp.send_folder from config)",
)
send_folder_parser.add_argument(
"--to",
type=str,
required=True,
help="Recipient email address",
default=None,
help="Recipient email (default: smtp.default_recipient from config)",
)
send_folder_parser.add_argument(
"--subject",
Expand Down Expand Up @@ -237,8 +239,8 @@ def parse_arguments() -> argparse.Namespace:
password_set_parser.add_argument(
"--user",
type=str,
required=True,
help="IMAP user login",
default=None,
help="IMAP user login (default: imap.user from config)",
)
password_set_parser.add_argument(
"--password-file",
Expand All @@ -261,8 +263,8 @@ def parse_arguments() -> argparse.Namespace:
password_clear_parser.add_argument(
"--user",
type=str,
required=True,
help="IMAP user login",
default=None,
help="IMAP user login (default: imap.user from config)",
)

# Command: config (with subcommands)
Expand Down
27 changes: 17 additions & 10 deletions email_processor/imap/auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
encrypt_password,
is_encrypted,
)
from email_processor.utils.redact import redact_email


def get_imap_password(imap_user: str, config_path: Optional[str] = None) -> str:
Expand All @@ -32,19 +33,21 @@ def get_imap_password(imap_user: str, config_path: Optional[str] = None) -> str:
try:
password = decrypt_password(stored_password, config_path)
logger.debug(
"password_decrypted_length", password_length=len(password), user=imap_user
"password_decrypted_length",
password_length=len(password),
user=redact_email(imap_user),
)
logger.info(
"password_retrieved_decrypted",
user=imap_user,
user=redact_email(imap_user),
service=KEYRING_SERVICE_NAME,
)
return password
except ValueError as e:
# Decryption failed - system characteristics may have changed
logger.error(
"password_decryption_failed",
user=imap_user,
user=redact_email(imap_user),
error=str(e),
hint="System characteristics may have changed. Password needs to be re-entered.",
)
Expand All @@ -56,7 +59,7 @@ def get_imap_password(imap_user: str, config_path: Optional[str] = None) -> str:
except Exception as e:
logger.error(
"password_decryption_error",
user=imap_user,
user=redact_email(imap_user),
error=str(e),
error_type=type(e).__name__,
)
Expand All @@ -68,13 +71,13 @@ def get_imap_password(imap_user: str, config_path: Optional[str] = None) -> str:
# Old format - unencrypted password
logger.info(
"password_retrieved",
user=imap_user,
user=redact_email(imap_user),
service=KEYRING_SERVICE_NAME,
encrypted=False,
)
return stored_password # type: ignore[no-any-return]

logger.info("password_not_found", user=imap_user)
logger.info("password_not_found", user=redact_email(imap_user))
pw = getpass.getpass(f"Enter IMAP password for {imap_user}: ")
if not pw:
raise ValueError("Password not entered, operation aborted.")
Expand All @@ -87,22 +90,26 @@ def get_imap_password(imap_user: str, config_path: Optional[str] = None) -> str:
keyring.set_password(KEYRING_SERVICE_NAME, imap_user, encrypted_password)
logger.info(
"password_saved_encrypted",
user=imap_user,
user=redact_email(imap_user),
service=KEYRING_SERVICE_NAME,
encrypted=True,
)
except Exception as e:
logger.error("password_save_error", user=imap_user, error=str(e))
logger.error("password_save_error", user=redact_email(imap_user), error=str(e))
# Try saving unencrypted as fallback
try:
keyring.set_password(KEYRING_SERVICE_NAME, imap_user, pw)
logger.warning(
"password_saved_unencrypted_fallback",
user=imap_user,
user=redact_email(imap_user),
error=str(e),
)
except Exception as e2:
logger.error("password_save_fallback_error", user=imap_user, error=str(e2))
logger.error(
"password_save_fallback_error",
user=redact_email(imap_user),
error=str(e2),
)

return pw

Expand Down
Loading
Loading