FTP Bridge v2.1.0 🌉

A professional secure bridge for integrating FTP/SFTP servers with Power BI and analytics systems.

📋 Table of Contents

Features
Security
Quick Start
Installation
Configuration
Usage
Administrative Utilities
Monitoring
Docker
Power BI Integration
Troubleshooting

🚀 Features

✨ New in v2.1.0

🔐 SFTP Host Key Verification: Host key verification for maximum security
📡 HEAD Endpoint: Retrieve file metadata without downloading (for Power BI preflight)
🛡️ Path Sanitizer: Advanced protection against path traversal attacks
🔒 PII Masking: Masking of personal data in logs (email, users)
⚡ Auto-Chunk Tuning: Automatic chunk size optimization for large files
🔶 Degraded Mode: Graceful startup without tokens with limited functionality
🏗️ Storage Backend Architecture: Unified architecture for FTP/FTPS/SFTP
🔧 Default Rate Limits: Built-in 100/minute limits for extra protection

✨ Features in v2.0.0

🔒 Enhanced Security: Tokens only via environment variables
⚡ Rate Limiting: Protection from DDoS and abuse with slowapi
🌐 Secure CORS: Configurable domains instead of wildcard *
🔐 SFTP Support: Encrypted file transfer via paramiko
📊 Log Rotation: Automatic log size management
⚙️ Pydantic v2: Modern configuration validation with BaseSettings
🛠️ Advanced Admin Utilities: Monitoring, diagnostics, maintenance

🌟 Core Features

Streaming large files without loading into memory
Protocol support: FTP, FTPS, SFTP
RESTful API with automatic documentation
File validation by extension and size
Path Traversal Protection
Detailed logging with rotation
Health Check endpoints for monitoring
Docker Ready with prebuilt containers

🔒 Security

Critical Security Improvements

❌ Fixed in v2.0.0:

✅ Hardcoded tokens removed from code
✅ Added rate limiting for attack protection
✅ CORS configured for specific domains (not *)
✅ Added SFTP support for encrypted transfer
✅ Implemented log rotation
✅ Modern secret management with Pydantic v2

Security Architecture

Token Management 🔑
- Tokens only via environment variables
- Minimum length 32 characters
- Automatic token strength validation
Rate Limiting ⚡
- Configurable request limits
- DDoS protection
- IP-based blocking
CORS Policy 🌐
- Domain restriction in production
- Configurable methods and headers
- Credentials support
File Validation 📁
- File extension checks
- Path traversal protection
- File size limits

🛡️ Security Fixes and Compliance

Summary of Security Fixes (v2.1.0)

Token Storage: All hardcoded tokens removed; tokens only from environment variables; minimum length 32 characters; automatic validation.
Rate Limiting: Added slowapi middleware; default 100/minute; configurable via environment.
CORS: Wildcard * removed; customizable via env; separate dev/prod configs; validation in monitoring.
SFTP/FTPS: Full SFTP support with host key verification; FTPS by default; warnings for insecure FTP.
Log Rotation: RotatingFileHandler with configurable size/count; log size monitoring.
Secrets Management: Full transition to Pydantic v2 BaseSettings; .env support; type-level validation.
Additional: Path sanitizer, PII masking in logs, degraded mode, HEAD endpoint, auto-chunk tuning, unified backend architecture.

Security Architecture (Multi-layered)

Authentication: Tokens only from environment, min 32 chars, auto validation.
Authorization: Bearer token, client identification, access logging.
Rate Limiting: DDoS protection, IP blocking, configurable limits.
CORS Policy: Domain restriction, secure headers, method control.
Data Validation: Pydantic v2, file checks, path traversal protection.
Encryption: SFTP/FTPS support, secure transfer.

Security Testing

Automated: Path traversal, token validation, rate limiting, CORS, SFTP, HEAD endpoint, degraded mode.
Manual: Penetration testing, brute force protection, file system boundaries, protocol validation.

Production Security Checklist

Tokens
- Generate unique tokens 32+ characters
- Set environment variables
- Remove all example tokens
CORS
- Specify exact domains (not "*")
- Test access from Power BI
Rate Limiting
- Set limits for your load
- Test attack protection
Protocols
- Enable FTPS or SFTP
- Test encrypted connections
Monitoring
- Set up health check
- Check log rotation

Additional Measures:

Regular token rotation (every 90 days)
Monitoring rate limiting and security
Use Docker Secrets in production
Log auditing for suspicious activity

Compliance Metrics

OWASP Top 10 2021:
- A01 Broken Access Control: Resolved by tokens/validation
- A02 Cryptographic Failures: Resolved by FTPS/SFTP
- A03 Injection: Resolved by path sanitizer
- A05 Security Misconfiguration: Resolved by Pydantic v2
- A07 Identification/Authentication Failures: Resolved by tokens
- A09 Security Logging: Resolved by PII masking
Security Standards:
- ISO 27001: Secrets management
- NIST Cybersecurity Framework: Multi-layered protection
- PCI DSS: Encrypted data transfer

🚀 Quick Start

1. Clone and Install

git clone https://github.com/your-repo/ftp_bridge.git
cd ftp_bridge
pip install -r requirements.txt

2. Set Up Security Tokens

# Generate tokens
python -c "import secrets; print('FTP_BRIDGE_TOKEN_POWERBI=' + secrets.token_hex(16))"
python -c "import secrets; print('FTP_BRIDGE_TOKEN_EXCEL=' + secrets.token_hex(16))"

# Create .env file
echo "FTP_BRIDGE_TOKEN_POWERBI=your_generated_token_here" > .env
echo "FTP_BRIDGE_TOKEN_EXCEL=another_generated_token_here" >> .env

3. Start with Auto-Setup

python start.py

4. Test the Service

curl -H "Authorization: Bearer your_token" \
     "http://localhost:8000/download?host=ftp.example.com&user=demo&password=password&path=/&file=test.txt"

📦 Installation

Requirements

Python 3.8+
FastAPI 0.104+
Pydantic v2

Automatic Installation

python start.py  # Will check and install dependencies automatically

Manual Installation

pip install -r requirements.txt

Optional Dependencies

# For SFTP support
pip install paramiko

# For production
pip install gunicorn

⚙️ Configuration

Environment Variables

🔑 Security Tokens (REQUIRED)

# Client tokens (minimum 32 characters)
FTP_BRIDGE_TOKEN_POWERBI=aabbccdd11223344556677889900aabbccddee
FTP_BRIDGE_TOKEN_EXCEL=1122334455667788990011223344556677889900
FTP_BRIDGE_TOKEN_ANALYTICS=9988776655443322110099887766554433221100

🌐 CORS and Security

# Allowed domains (DO NOT use * in production!)
FTP_BRIDGE_CORS_ORIGINS=https://app.powerbi.com,https://your-domain.com

# Rate limiting
FTP_BRIDGE_RATE_LIMIT_ENABLED=true
FTP_BRIDGE_RATE_LIMIT_REQUESTS=100      # Requests per hour
FTP_BRIDGE_RATE_LIMIT_WINDOW=3600       # Window in seconds

🔌 Protocols

# Default protocol (ftps recommended for security)
FTP_BRIDGE_DEFAULT_PROTOCOL=ftps        # ftp, ftps, sftp

# FTPS settings
FTP_BRIDGE_USE_FTPS=true

# Ports
FTP_BRIDGE_FTP_PORT=21                  # FTP port
FTP_BRIDGE_SFTP_PORT=22                 # SFTP port

# SFTP Host Key Verification (for maximum security)
FTP_BRIDGE_KNOWN_HOSTS_PATH=/home/user/.ssh/known_hosts

# Degraded mode (graceful fallback)
FTP_BRIDGE_DEGRADED_MODE=false

📊 Log Rotation

FTP_BRIDGE_LOG_ROTATION=true
FTP_BRIDGE_LOG_MAX_SIZE=10485760        # 10MB
FTP_BRIDGE_LOG_BACKUP_COUNT=5

Create .env File

# Copy from template
cp env_example.txt .env

# Edit tokens
nano .env

🔧 Usage

API Endpoints

File Download

GET /download?host=ftp.example.com&user=demo&password=password&path=/reports/data.xlsx&protocol=ftps
Authorization: Bearer your_token_here

File Metadata (HEAD) - New in v2.1.0

HEAD /download?host=ftp.example.com&user=demo&password=password&path=/reports/data.xlsx&protocol=ftps
Authorization: Bearer your_token_here

Returns headers:

X-File-Size - File size in bytes
X-Protocol - Protocol used (ftp/ftps/sftp)
X-File-Name - File name
Content-Length - Download size

Used by Power BI for preflight requests and optimized loading

Parameters

host - FTP/SFTP server
user - Username
password - Password
path - Full file path on the server (including file name)
protocol - auto, ftp, ftps, sftp (optional, default is ftps)

Health Check

GET /health

Advanced diagnostics v2.1.0:

{
  "status": "healthy",
  "degraded_mode": false,
  "default_protocol": "ftps",
  "sftp_host_key_verification": true,
  "protocols_available": ["ftp", "ftps", "sftp"]
}

Documentation

Swagger UI: http://localhost:8000/docs
ReDoc: http://localhost:8000/redoc

Integration Examples

Python requests

import requests

headers = {"Authorization": "Bearer your_token"}
params = {
    "host": "ftp.example.com",
    "user": "demo", 
    "password": "password",
    "path": "/data/report.xlsx",  # Full file path
    "protocol": "ftps"  # Secure FTPS by default
}

# First, get metadata (optional)
head_response = requests.head("http://localhost:8000/download", 
                             headers=headers, params=params)
file_size = head_response.headers.get('X-File-Size')
print(f"File size: {file_size} bytes")

# Download file
response = requests.get("http://localhost:8000/download", 
                       headers=headers, params=params, stream=True)

with open("downloaded_file.xlsx", "wb") as f:
    for chunk in response.iter_content(chunk_size=8192):
        f.write(chunk)

cURL examples

# HEAD request for metadata
curl -I -H "Authorization: Bearer your_token" \
     "http://localhost:8000/download?host=ftp.example.com&user=demo&password=password&path=/data/report.xlsx&protocol=ftps"

# Download file
curl -H "Authorization: Bearer your_token" \
     -o downloaded_file.xlsx \
     "http://localhost:8000/download?host=ftp.example.com&user=demo&password=password&path=/data/report.xlsx&protocol=ftps"

New in v2.1.0

# Check degraded mode
curl "http://localhost:8000/health" | jq '.degraded_mode'

# Auto-optimization for large files
curl -H "Authorization: Bearer your_token" \
     -I "http://localhost:8000/download?path=/large_file.zip" \
     | grep "X-Auto-Chunk-Tuned"

🛠️ Administrative Utilities

Token Management

# List tokens with security analysis
python admin_utils.py tokens list

# Generate new token
python admin_utils.py tokens generate --length 32 --client "Power BI"

# Validate token
python admin_utils.py tokens validate your_token_here

System Monitoring

# Full system diagnostics
python admin_utils.py monitor health

# JSON output for automation
python admin_utils.py monitor health --json

Maintenance

# Clean up old temp files
python admin_utils.py maintenance cleanup --max-age-hours 24

# Validate configuration
python admin_utils.py maintenance validate

# Backup configuration
python admin_utils.py maintenance backup --dir backups

📊 Monitoring

Health Check

{
  "status": "healthy",
  "temp_dir": "./temp",
  "temp_dir_exists": true,
  "temp_dir_writable": true,
  "active_tokens": 3,
  "protocols_available": ["ftp", "ftps", "sftp"],
  "rate_limit_enabled": true
}

System Checks

✅ Temp directory status
✅ Log rotation configuration
✅ Settings validation
✅ Token security analysis
✅ Performance settings

Testing

# Comprehensive testing
python test_api.py

# Tests include:
# - Basic endpoints
# - Authentication and tokens
# - Parameter validation
# - Rate limiting
# - CORS headers
# - SFTP support
# - Real file downloads

🐳 Docker

Quick Start

# Build and run
docker-compose up -d

# Set tokens via environment
docker-compose exec ftp-bridge python admin_utils.py tokens generate

Production Setup

# docker-compose.prod.yml
services:
  ftp-bridge:
    environment:
      - FTP_BRIDGE_TOKEN_POWERBI=${POWERBI_TOKEN}
      - FTP_BRIDGE_CORS_ORIGINS=https://app.powerbi.com
      - FTP_BRIDGE_RATE_LIMIT_REQUESTS=50
      - FTP_BRIDGE_USE_FTPS=true

📈 Power BI Integration

Power Query Example

let
    // Settings
    ApiUrl = "http://your-server:8000/download",
    ApiToken = "your_secure_token_here",
    
    // FTP Parameters
    FtpHost = "ftp.your-company.com",
    FtpUser = "your_user",
    FtpPassword = "your_password",
    FilePath = "/reports/monthly_data.xlsx",
    
    // Build URL
    Url = ApiUrl & "?" & 
          "host=" & FtpHost & 
          "&user=" & FtpUser & 
          "&password=" & FtpPassword & 
          "&path=" & FilePath & 
          "&file=data.xlsx" &
          "&protocol=sftp",
    
    // Authorization headers
    Headers = [
        #"Authorization" = "Bearer " & ApiToken,
        #"Content-Type" = "application/octet-stream"
    ],
    
    // Download file
    Response = Web.Contents(Url, [Headers=Headers]),
    
    // Process Excel file
    Excel = Excel.Workbook(Response),
    Sheet = Excel[Data]{[Item="Sheet1",Kind="Sheet"]}[Data]
in
    Sheet

Power BI Service Setup

Add domain to CORS:

FTP_BRIDGE_CORS_ORIGINS=https://app.powerbi.com,https://msit.powerbi.com

Use HTTPS in production
Set up auto-refresh with appropriate limits

🚨 Troubleshooting

Security Issues

Tokens not set

❌ CRITICAL ERROR: No tokens found!

# Solution:
export FTP_BRIDGE_TOKEN_CLIENT1=$(python -c 'import secrets; print(secrets.token_hex(16))')

CORS errors in browser

⚠️ CORS allows any domains in production

# Solution:
FTP_BRIDGE_CORS_ORIGINS=https://app.powerbi.com,https://your-domain.com

Rate limiting triggered

{"detail": "Rate limit exceeded: 100 per 3600 seconds"}

Solution: Increase limits or optimize request frequency

Connection Issues

SFTP unavailable

⚠️ SFTP support unavailable. Install paramiko.

# Solution:
pip install paramiko

Port issues

❌ Port 8000 already in use

# Solution:
FTP_BRIDGE_PORT=8001

Diagnostics

# Full diagnostics
python admin_utils.py monitor health

# Check configuration
python admin_utils.py maintenance validate

# API testing
python test_api.py

📄 License

MIT License - see LICENSE for details

🤝 Support

Documentation: API Docs
Issues: GitHub Issues
Security: Report vulnerabilities via private channels

🔄 Changelog

v2.0.0 - Security and Performance

✨ Pydantic v2 BaseSettings for configuration
🔒 Hardcoded tokens removed
⚡ Rate limiting with slowapi
🌐 Secure CORS for production
🔐 SFTP support via paramiko
📊 Log rotation with RotatingFileHandler
🛠️ Advanced admin utilities
🧪 Comprehensive security testing

v1.0.0 - First Release

🌉 Basic FTP Bridge functionality
📡 File streaming via FastAPI
🔑 Simple token authentication
📖 Automatic API documentation

FTP Bridge v2.0.0 - A professional solution for secure FTP/SFTP integration with analytics systems 🚀

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
.github/workflows		.github/workflows
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
admin_utils.py		admin_utils.py
config.py		config.py
docker-compose.yml		docker-compose.yml
env_example.txt		env_example.txt
environment.example		environment.example
main.py		main.py
requirements.txt		requirements.txt
start.py		start.py
storage_backend.py		storage_backend.py
test_api.py		test_api.py

Folders and files

Latest commit

History

Repository files navigation

FTP Bridge v2.1.0 🌉

📋 Table of Contents

🚀 Features

✨ New in v2.1.0

✨ Features in v2.0.0

🌟 Core Features

🔒 Security

Critical Security Improvements

Security Architecture

🛡️ Security Fixes and Compliance

Summary of Security Fixes (v2.1.0)

Security Architecture (Multi-layered)

Security Testing

Production Security Checklist

Compliance Metrics

🚀 Quick Start

1. Clone and Install

2. Set Up Security Tokens

3. Start with Auto-Setup

4. Test the Service

📦 Installation

Requirements

Automatic Installation

Manual Installation

Optional Dependencies

⚙️ Configuration

Environment Variables

🔑 Security Tokens (REQUIRED)

🌐 CORS and Security

🔌 Protocols

📊 Log Rotation

Create .env File

🔧 Usage

API Endpoints

File Download

File Metadata (HEAD) - New in v2.1.0

Parameters

Health Check

Documentation

Integration Examples

Python requests

cURL examples

New in v2.1.0

🛠️ Administrative Utilities

Token Management

System Monitoring

Maintenance

📊 Monitoring

Health Check

System Checks

Testing

🐳 Docker

Quick Start

Production Setup

📈 Power BI Integration

Power Query Example

Power BI Service Setup

🚨 Troubleshooting

Security Issues

Tokens not set

CORS errors in browser

Rate limiting triggered

Connection Issues

SFTP unavailable

Port issues

Diagnostics

📄 License

🤝 Support

🔄 Changelog

v2.0.0 - Security and Performance

v1.0.0 - First Release

About

Resources

Stars

Watchers

Forks