Update from head

.github/copilot-instructions.md

@@ -0,0 +1,130 @@
+# SmartSpin2k - ESP32 Smart Trainer Firmware
+
+SmartSpin2k is an ESP32-based DIY smart trainer project that converts any spin bike into a connected fitness device compatible with Zwift, TrainerRoad, and other training apps. The firmware controls stepper motor resistance, handles BLE communication, serves a web interface, and manages sensor data.
+
+Always reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here.
+
+## Working Effectively
+
+### Bootstrap Environment
+Install required tools and dependencies:
+- `sudo apt-get update && sudo apt-get install -y build-essential git python3 python3-pip`
+- `pip install platformio pre-commit`
+- `pre-commit install --hook-type pre-push`
+
+### Build the Firmware
+- **CRITICAL**: Build takes 15-45 minutes depending on network connectivity. NEVER CANCEL. Set timeout to 60+ minutes.
+- `pio run --environment release` -- builds ESP32 firmware. NEVER CANCEL: Build takes 15-45 minutes on first run due to platform/toolchain downloads.
+- `pio run --target buildfs` -- builds filesystem. Takes 2-5 minutes.
+- **Network Issues**: If platform downloads fail with HTTPClientError, this is due to firewall/network restrictions. The build cannot proceed without internet access to download ESP32 toolchain.
+
+### Testing
+- **CRITICAL**: Native tests take 5-15 minutes. NEVER CANCEL. Set timeout to 30+ minutes.
+- `pio test --environment native` -- runs unit tests using Unity framework. NEVER CANCEL: Takes 5-15 minutes on first run.
+- Tests validate sensor data parsing, power calculations, BLE communication, and stepper motor control.
+- **Network Issues**: Native platform download may fail with HTTPClientError due to firewall restrictions.
+
+### Code Quality and Validation
+- `pre-commit run --all-files` -- runs license header insertion. Takes 1-2 minutes.
+- `pio check -e debug` -- runs cppcheck static analysis on debug environment. Takes 2-5 minutes. **Network Issues**: May fail with HTTPClientError due to platform download restrictions.
+- Python build scripts (always work):
+  - `python git_tag_macro.py` -- generates firmware version from git tags
+  - `python build_date_macro.py` -- generates build timestamp  
+  - `python cert_updater.py` -- updates SSL certificates (may fail with network issues)
+
+### Run the Application
+- **Build First**: Always complete the bootstrap and build steps before attempting to run.
+- The application runs on ESP32 hardware - cannot be executed in the sandbox environment.
+- Web interface available at device IP on port 80 when running on hardware.
+- BLE services broadcast as "SmartSpin2k" when running on hardware.
+
+## Validation
+
+### Manual Testing Scenarios
+After making code changes, always validate:
+1. **Build Validation**: Ensure `pio run --environment release` completes successfully.
+2. **Test Validation**: Ensure `pio test --environment native` passes all Unity tests.
+3. **Code Quality**: Run `pre-commit run --all-files` and fix any license header issues.
+4. **BLE Service Changes**: When modifying BLE services, verify characteristic UUIDs match the CustomCharacteristic.md specification.
+5. **Power Calculations**: When changing power table or ERG mode code, run tests in test_pt_lookup_*.cpp files.
+6. **Sensor Data**: When modifying sensor parsing, validate with tests in test_*Data.cpp files.
+
+### Critical Areas to Test
+- **Power Table**: Always validate power lookup and resistance calculations after changes to Power_Table.cpp or PowerTable_Helpers.cpp
+- **BLE Services**: Test characteristic read/write operations when modifying BLE_*_Service.cpp files
+- **ERG Mode**: Validate resistance control when changing ERG_Mode.cpp
+- **Stepper Control**: Test motor control when modifying stepper-related code in Main.cpp
+
+## Common Tasks
+
+### Repository Structure
+Key directories and their purpose:
+```
+/src                 -- Main ESP32 firmware source code
+/lib/SS2K/src        -- Core library with sensor parsing and data structures
+/include             -- Header files and configuration
+/test                -- Unity unit tests for native environment
+/data                -- Web interface HTML/CSS files
+/Hardware            -- 3D printing files and PCB designs
+/.github/workflows   -- CI/CD pipeline definitions
+```
+
+### Important Files
+- `platformio.ini` -- Build configuration for ESP32 and native environments
+- `include/settings.h` -- Hardware pin definitions and configuration constants
+- `CustomCharacteristic.md` -- BLE characteristic specification and usage
+- `src/Main.cpp` -- Main firmware entry point and setup
+- `lib/SS2K/src/sensors/` -- Sensor data parsing classes
+
+### Build Dependencies
+External libraries loaded automatically by PlatformIO:
+- NimBLE-ESP32 for Bluetooth Low Energy
+- TMCStepper for stepper motor control
+- FastAccelStepper for smooth motor movement
+- ArduinoJson for configuration and web API
+- ArduinoWebsockets for real-time web communication
+
+### Configuration
+- Default device name: "SmartSpin2k"
+- Default WiFi password: "password"
+- Web interface served on port 80
+- BLE service UUID: "77776277-7877-7774-4466-896665500000"
+- Over-the-air update URL: configured in settings.h
+
+### Hardware Compatibility
+- ESP32 DevKit v1 board (primary target)
+- TMC2209 stepper motor driver
+- Custom PCB designs in Hardware/ directory
+- Support for multiple bike mount configurations
+
+### Known Issues and Limitations
+- **Network Connectivity**: Platform and toolchain downloads may fail due to firewall restrictions. All build commands (`pio run`, `pio test`, `pio check`) require internet access on first run.
+- **SSL Certificates**: cert_updater.py may fail to fetch current certificates due to network restrictions
+- **Hardware Testing**: Cannot test actual motor control or BLE communication without physical hardware
+- **Build Times**: Initial builds require internet access and take 15-45 minutes due to large platform downloads
+
+### Troubleshooting Common Issues
+- **HTTPClientError during build**: This indicates network/firewall restrictions preventing platform downloads. No workaround available in restricted environments.
+- **Platform not found**: Run `pio platform install espressif32` to manually install the ESP32 platform (requires internet).
+- **Test failures**: Ensure you're running tests in native environment: `pio test -e native`
+- **SSL certificate warnings**: Update certificates with `python cert_updater.py` or manually update `include/cert.h`
+- **Build flag errors**: The Python scripts in build_flags must execute successfully. Test them individually if build fails.
+
+### Environment Verification
+Before working on the project, verify your environment:
+```bash
+# Check tools are installed
+which python3 pio pre-commit
+# Verify project configuration
+pio project config
+# Test build scripts
+python git_tag_macro.py && python build_date_macro.py
+```
+
+### Debugging Tips
+- Use `pio device monitor` to view serial output when connected to ESP32 hardware
+- Check `include/cert.h` if experiencing SSL errors during firmware updates
+- Monitor memory usage with DEBUG_STACK enabled in settings.h
+- BLE debugging available through web interface at `/develop.html`
+
+Always run `pre-commit run --all-files` before completing changes to ensure code meets project standards.

.github/workflows/build.yml

@@ -9,19 +9,29 @@ jobs:
   build:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v5
         with:
           fetch-depth: 0
+      - name: Cache PlatformIO
+        uses: actions/cache@v4
+        with:
+          path: ~/.platformio
+          key: ${{ runner.os }}-platformio-${{ hashFiles('platformio.ini') }}
+          restore-keys: |
+            ${{ runner.os }}-platformio-
       - name: Ensure changelog is updated
         uses: dangoslen/changelog-enforcer@v3.6.1
       - name: Set up python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v6
         with:
-          python-version: '3.x'
+          python-version: '3.11'
           architecture: 'x64'
+      - name: Upgrade pip tooling
+        run: |
+          python -m pip install --upgrade pip setuptools wheel
       - name: Get current date
         id: date
-        run: echo "::set-output name=date::$(date +'%y.%-m.%-d')"
+        run: echo "date=$(date +'%y.%-m.%-d')" >> $GITHUB_OUTPUT
       - name: Test with environment variables
         run: echo $TAG_NAME - $RELEASE_NAME
         env:
@@ -43,12 +53,32 @@ jobs:
         run: python --version
       - name: Install PlatformIO
         run: python -m pip install platformio
+      - name: Pre-install PlatformIO packages (release env)
+        run: platformio pkg install -e release
+      - name: Ensure ESP-IDF component manager is available
+        shell: bash
+        run: |
+          set -euo pipefail
+          PIO_HOME="$HOME/.platformio"
+          # Try to locate PlatformIO-managed ESP-IDF virtualenv
+          ESPIDF_VENV=$(ls -d "$PIO_HOME/penv/.espidf-"* 2>/dev/null | head -n 1 || true)
+          if [ -z "$ESPIDF_VENV" ]; then
+            # Trigger framework fetch which usually creates the venv
+            platformio platform show espressif32 || true
+            ESPIDF_VENV=$(ls -d "$PIO_HOME/penv/.espidf-"* 2>/dev/null | head -n 1 || true)
+          fi
+          if [ -n "$ESPIDF_VENV" ] && [ -x "$ESPIDF_VENV/bin/python" ]; then
+            "$ESPIDF_VENV/bin/python" -m pip install --upgrade pip setuptools wheel
+            "$ESPIDF_VENV/bin/python" -m pip install --upgrade idf-component-manager
+          else
+            echo "ESP-IDF venv not found yet; proceeding. PlatformIO should install deps during build."
+          fi
       - name: Check pre-commit hooks
         uses: pre-commit/action@v3.0.1
       - name: Build firmware
-        run: platformio run
+        run: platformio run -e release
       - name: Build filesystem
-        run: platformio run --target buildfs
+        run: platformio run -e release --target buildfs
       - name: Create artifacts
         run: |
           7z a SmartSpin2kFirmware-${{ steps.date.outputs.date }}.bin.zip ./.pio/build/release/*.bin
@@ -58,12 +88,12 @@ jobs:
           name: all-artifacts
           path: SmartSpin2kFirmware-${{ steps.date.outputs.date }}.bin.zip
       - name: Download Artifacts
-        uses: actions/download-artifact@v4
+        uses: actions/download-artifact@v5
         with:
           name: all-artifacts
       - name: Get tag info
         id: tag_info
-        run: echo ::set-output name=SOURCE_TAG::${{ steps.date.outputs.date }}
+        run: echo "SOURCE_TAG=${{ steps.date.outputs.date }}" >> $GITHUB_OUTPUT
       - name: Create release
         uses: softprops/action-gh-release@v2
         env:

.github/workflows/copilot-setup-steps.yml

@@ -0,0 +1,92 @@
+name: "Copilot Setup Steps"
+
+on:
+  workflow_dispatch:
+  push:
+    paths:
+      - .github/workflows/copilot-setup-steps.yml
+  pull_request:
+    paths:
+      - .github/workflows/copilot-setup-steps.yml
+
+jobs:
+  # IMPORTANT: Job name must be exactly this.
+  copilot-setup-steps:
+    # You can upgrade to a larger Ubuntu runner label later if needed (e.g. ubuntu-4-core)
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+
+    # Minimize permissions; Copilot will get its own token later.
+    permissions:
+      contents: read
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v5
+        with:
+          # fetch-depth overridden internally for Copilot anyway; keep explicit for normal runs
+          fetch-depth: 0
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.x'
+
+      - name: Cache PlatformIO core and packages
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.platformio
+          key: pio-${{ runner.os }}-${{ hashFiles('platformio.ini', 'dependencies.lock') }}
+          restore-keys: |
+            pio-${{ runner.os }}-
+
+      - name: Install PlatformIO Core
+        run: |
+          python -m pip install --upgrade pip
+            pip install --upgrade platformio
+
+      - name: Show PlatformIO info
+        run: |
+          platformio --version
+          platformio system info
+
+      - name: Pre-download native test dependencies
+        run: |
+          platformio pkg install -e native || echo "Native env install attempted"
+
+      - name: Pre-download ESP32 toolchains and libraries (release env)
+        run: |
+          # Only install packages (faster + ensures offline availability later)
+          platformio pkg install -e release
+
+      - name: Warm build cache (lightweight compile) for release env
+        run: |
+          # A full build ensures toolchains, frameworks, and libs are all present
+          # If this becomes too slow, you can remove this step and rely on pkg install only.
+          platformio run -e release || exit 0
+
+      - name: Warm build cache for native env (tests)
+        run: |
+          platformio run -e native || exit 0
+
+      - name: Validate native tests (non-blocking)
+        run: |
+          set +e
+          platformio test -e native
+          # Do not fail setup if tests fail; Copilot will handle fixes.
+          exit 0
+
+      - name: Summarize cached packages
+        run: |
+          du -sh ~/.platformio/packages/* || true
+          du -sh ~/.platformio/platforms/* || true
+
+      - name: Guidance for future adjustments
+        shell: bash
+        run: |
+          echo "Setup steps completed. You can:"
+          echo " - Upgrade runner: change runs-on to ubuntu-4-core if builds are slow." \
+               "\n - Enable LFS: add 'with: lfs: true' to checkout step if using Git LFS." \
+               "\n - Trim steps: remove warm build steps if time exceeds limits." \
+               "\n - Add more pkg installs for other environments if created later."

.github/workflows/pre-commit.yml

@@ -7,7 +7,7 @@ jobs:
   pre-commit:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v5
         with:
           fetch-depth: 0
       - uses: dorny/paths-filter@v3
@@ -21,6 +21,6 @@ jobs:
       - name: Ensure changelog is updated
         uses: dangoslen/changelog-enforcer@v3.6.1
       - name: Setup python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v6
       - name: Check pre-commit hooks
         uses: pre-commit/action@v3.0.1

.github/workflows/update-changelog.yml

@@ -0,0 +1,63 @@
+name: Update Changelog
+
+on:
+  pull_request:
+    branches:
+      - develop
+    types: [opened, synchronize, reopened]
+
+jobs:
+  update-changelog:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+      pull-requests: write
+
+    steps:
+      - uses: actions/checkout@v5
+        with:
+          fetch-depth: 0
+          ref: ${{ github.event.pull_request.head.ref }}
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.x'
+
+      - name: Update Changelog
+        run: |
+          LATEST_TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "0.0.0")
+          echo "Latest tag: $LATEST_TAG"
+
+          if ! grep -q "## \[Unreleased\]" CHANGELOG.md; then
+            echo "No [Unreleased] section found"
+            exit 0
+          fi
+
+          FIRST_VERSION=$(grep -oP "## \[\K[0-9]+\.[0-9]+\.[0-9]+" CHANGELOG.md | head -1 || echo "0.0.0")
+
+          if [ "$LATEST_TAG" = "$FIRST_VERSION" ]; then
+            echo "Latest tag matches first version in changelog"
+            exit 0
+          fi
+
+          # Create temporary files
+          echo -e "# Changelog\n\nAll notable changes to this project will be documented in this file.\n\nThe format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)\nand this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).\n\n## [Unreleased]\n\n### Added\n\n### Changed\n\n### Hardware\n" > header.tmp
+
+          # Update the old unreleased section
+          sed "0,/## \[Unreleased\]/s/## \[Unreleased\]/## [$LATEST_TAG]/" CHANGELOG.md | tail -n +7 > content.tmp
+
+          # Combine files
+          cat header.tmp content.tmp > CHANGELOG.md
+          rm header.tmp content.tmp
+
+          if git diff --quiet CHANGELOG.md; then
+            echo "No changes to commit"
+            exit 0
+          fi
+
+          git config --local user.email "action@github.com"
+          git config --local user.name "GitHub Action"
+          git add CHANGELOG.md
+          git commit -m "Update changelog for version $LATEST_TAG"
+          git push