Skip blank rows and section headers in Excel processing

AGENTS.md

@@ -0,0 +1,85 @@
+# QuestionnaireAgent_v3 Development Guidelines
+
+Auto-generated from all feature plans. Last updated: 2025-10-09
+
+## Virtual Environment
+
+**IMPORTANT**: All development and testing work must be done in the `.venv312` virtual environment.
+
+### Activating the Virtual Environment
+
+```bash
+source .venv312/bin/activate
+```
+
+### Running Tests
+
+Always activate the virtual environment before running tests:
+
+```bash
+source .venv312/bin/activate
+python -m pytest tests/unit/test_name.py -v
+```
+
+### Running the Application
+
+```bash
+source .venv312/bin/activate
+python src/main.py
+```
+
+## Active Technologies
+- Python 3.11+ + agent-framework-azure-ai (--pre), azure-ai-projects, azure-identity, tkinter, openpyxl, pandas, playwright, pytest (001-rewrite-questionnaire-agent)
+- Python 3.11+ + tkinter (GUI), openpyxl (Excel), agent-framework-azure-ai (multi-agent), azure-ai-projects, azure-identity (002-amend-the-answer)
+- Excel files (.xlsx) on local filesystem (002-amend-the-answer)
+- Python 3.11+ + FastAPI (web server), Uvicorn (ASGI server), Jinja2 (templates), ag-Grid Community Edition or Handsontable (spreadsheet), Playwright (testing) (004-add-web-mode)
+- Server-side session storage (in-memory dict with session IDs), temporary file storage for uploaded Excel files (004-add-web-mode)
+
+## Project Structure
+```
+src/
+tests/
+```
+
+## Commands
+cd src; pytest; ruff check .
+
+## Code Style
+Python 3.11+: Follow standard conventions
+
+## Recent Changes
+- 004-add-web-mode: Added Python 3.11+ + FastAPI (web server), Uvicorn (ASGI server), Jinja2 (templates), ag-Grid Community Edition or Handsontable (spreadsheet), Playwright (testing)
+- 002-amend-the-answer: Added Python 3.11+ + tkinter (GUI), openpyxl (Excel), agent-framework-azure-ai (multi-agent), azure-ai-projects, azure-identity
+- 001-rewrite-questionnaire-agent: Added Python 3.11+ + agent-framework-azure-ai (--pre), azure-ai-projects, azure-identity, tkinter, openpyxl, pandas, playwright, pytest
+
+---
+
+## Running the Web Application
+
+### For Automated Testing (Playwright, CI/CD)
+
+**Always use `--no-browser` when running Playwright tests or any automated browser testing:**
+
+```bash
+# Correct: Start server without opening a browser tab
+python run_app.py --web --port 8080 --no-browser
+
+# Incorrect: This will pop up a browser tab on your default browser
+python run_app.py --web --port 8080
+```
+
+**Why this matters:**
+- When an AI agent runs Playwright or automated tests, it controls its own browser instance
+- Without `--no-browser`, the app will open a tab in the user's default browser every time
+- This causes random browser popups during test runs, which is disruptive
+- The Playwright-controlled browser is separate from the user's browser
+
+**Rule:** If you are writing or running automated tests that involve the web interface, always include `--no-browser` in the server start command.
+
+### For Manual Testing
+
+When you want to manually interact with the UI:
+```bash
+python run_app.py --web --port 8080
+```
+This will start the server AND open a browser tab automatically.

MIGRATION_NOTES.md

@@ -0,0 +1,59 @@
+# agent-framework rc5 Migration – Complete
+
+## Root Cause
+
+Two issues combined:
+
+1. The `agent-framework-azure-ai` package was upgraded from `1.0.0b260107` → `1.0.0rc5`, requiring API migration across the codebase.
+2. The `AzureAIAgentClient` (Foundry Agent Service v1) uses the `/assistants` API endpoint, which returns empty HTTP responses for `gpt-4.1` deployments — causing `JSONDecodeError: Expecting value: line 1 column 1 (char 0)`.
+
+## Fix Applied
+
+Migrated to **Foundry Agent Service v2** (`AzureAIClient`) which uses the `/agents` API endpoint, plus all rc5 API changes.
+
+**Packages installed:**
+- `agent-framework-azure-ai == 1.0.0rc5`
+- `agent-framework-core == 1.0.0rc5`
+- `azure-ai-agents == 1.2.0b5`
+
+## API Changes Applied
+
+| Old (b260107) | New (rc5) |
+|-----|-----|
+| `from agent_framework_azure_ai import AzureAIAgentClient` | `from agent_framework.azure import AzureAIClient` |
+| `AzureAIAgentClient(project_endpoint=..., credential=...)` | `AzureAIClient(project_endpoint=..., credential=..., model_deployment_name=...)` |
+| `ChatAgent(chat_client=client, name=..., instructions=...)` | `RawAgent(client, name=..., instructions=...)` |
+| `ChatMessage(role=Role.USER, text=...)` | `Message(role="user", text=...)` |
+| `Role.USER` (enum) | `"user"` (string literal) |
+| `WorkflowBuilder().set_start_executor(x).build()` | `WorkflowBuilder(start_executor=x).build()` |
+| `workflow.run_stream(data)` | `workflow.run(data, stream=True)` |
+| Agent names with spaces: `"Question Answerer"` | Kebab-case names: `"question-answerer"` |
+
+**Note:** `Role` changed from an enum to a `NewType` — use string literals `"user"` instead.
+
+**Note:** Agent names must be alphanumeric with hyphens, max 63 chars (Foundry v2 requirement).
+
+## Files Changed
+
+- `src/utils/azure_auth.py` — `AzureAIAgentClient` → `AzureAIClient` (import, type hints, constructors)
+- `src/agents/workflow_manager.py` — `AzureAIClient` import, `WorkflowBuilder(start_executor=...)`, `run(stream=True)`
+- `src/agents/question_answerer.py` — `RawAgent`, `Message`, `"user"` role, kebab-case name
+- `src/agents/answer_checker.py` — `RawAgent`, `Message`, `"user"` role, kebab-case name
+- `src/agents/link_checker.py` — `RawAgent`, `Message`, `"user"` role, kebab-case name
+- `src/excel/loader.py` — `AzureAIClient` import, kebab-case agent names
+- `src/excel/column_identifier.py` — `"user"` role string
+- `src/web/app.py` — `RawAgent` import, kebab-case agent name
+- `tests/unit/test_column_identifier.py` — Updated mock patches for `RawAgent`/`Message`
+
+## Current Status
+
+✅ **All migration changes applied and verified.**
+
+- Live test (`tests/test_live_azure_question.py`) passes end-to-end
+- 123 unit tests pass; 21 have pre-existing failures unrelated to this migration
+
+## How to Run
+
+```bash
+source .venv312/bin/activate && python run_app.py
+```

run_app.py

@@ -5,6 +5,10 @@
 import os
 import asyncio
 import argparse
+import threading
+import time
+import webbrowser
+import socket
 from pathlib import Path
 
 # Add the src directory to Python path
@@ -122,24 +126,211 @@ def parse_arguments():
         default=None,
         help="Path to Excel spreadsheet to process immediately after initialization"
     )
-
+
+    parser.add_argument(
+        "--web",
+        action="store_true",
+        help="Launch in web interface mode instead of desktop GUI"
+    )
+
+    parser.add_argument(
+        "--port",
+        type=int,
+        default=8080,
+        help="Port for web server (default: 8080, only used with --web)"
+    )
+
+    parser.add_argument(
+        "--no-browser",
+        action="store_true",
+        help="Don't automatically open browser (only used with --web)"
+    )
+
+    parser.add_argument(
+        "--mockagents",
+        action="store_true",
+        help="Use mock agents for testing (only used with --web)"
+    )
+
     return parser.parse_args()
 
 
+def is_port_in_use(port: int) -> bool:
+    """Check if a port is already in use.
+
+    Args:
+        port: Port number to check
+
+    Returns:
+        True if port is in use, False otherwise
+    """
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+        try:
+            s.bind(("127.0.0.1", port))
+            return False
+        except socket.error:
+            return True
+
+
+def wait_for_server(port: int, timeout: float = 10.0) -> bool:
+    """Wait for the web server to become available.
+
+    Args:
+        port: Port to check
+        timeout: Maximum time to wait in seconds
+
+    Returns:
+        True if server is available, False if timeout
+    """
+    import requests
+    start_time = time.time()
+    url = f"http://127.0.0.1:{port}/health"
+
+    while time.time() - start_time < timeout:
+        try:
+            response = requests.get(url, timeout=1)
+            if response.status_code == 200:
+                return True
+        except requests.exceptions.RequestException:
+            pass
+        time.sleep(0.5)
+
+    return False
+
+
+async def initialize_and_run_web(args):
+    """Initialize and run the web interface.
+
+    Args:
+        args: Parsed command line arguments
+
+    Returns:
+        Exit code (0 for success)
+    """
+    from utils.azure_auth import test_authentication
+    from utils.config import config_manager
+    from utils.exceptions import AuthenticationError, ConfigurationError
+
+    try:
+        # Step 1: Check port availability
+        if is_port_in_use(args.port):
+            print(f"\n❌ Port {args.port} is already in use.")
+            print(f"   Try a different port with: python run_app.py --web --port {args.port + 1}")
+            return 1
+
+        # Step 2: Validate configuration
+        print("Validating configuration...")
+        validation_result = config_manager.validate_configuration()
+
+        if not validation_result.is_valid:
+            error_details = "; ".join(validation_result.error_details)
+            print(f"\n❌ Configuration Error: {error_details}")
+            print("\nPlease check:")
+            print("1. .env file exists and contains required values")
+            print("2. AZURE_OPENAI_ENDPOINT is set")
+            print("3. AZURE_OPENAI_MODEL_DEPLOYMENT is set")
+            print("4. BING_CONNECTION_ID is set")
+            return 1
+
+        print("✓ Configuration validated successfully")
+
+        # Step 3: Test Azure authentication (skip if using mock agents)
+        if not args.mockagents:
+            print("\nTesting Azure authentication...")
+            print("(Checking for existing 'az login' or 'azd login' session)")
+
+            try:
+                await test_authentication()
+                print("✓ Azure authentication successful")
+            except AuthenticationError as e:
+                print(f"\n❌ Authentication Error: {e}")
+                return 1
+        else:
+            print("\nSkipping Azure authentication (mock agents mode)")
+
+        # Step 4: Start web server
+        print(f"\nStarting web server on http://127.0.0.1:{args.port}")
+        if args.mockagents:
+            print("*** MOCK AGENTS ENABLED - Using mock agents for testing ***")
+
+        from web.app import run_server, cleanup, set_mock_agents_mode
+
+        # Set mock agents mode if requested
+        if args.mockagents:
+            set_mock_agents_mode(True)
+
+        # Start server in background thread
+        server_thread = threading.Thread(
+            target=run_server,
+            kwargs={"host": "127.0.0.1", "port": args.port, "log_level": "warning"},
+            daemon=True
+        )
+        server_thread.start()
+
+        # Wait for server to be ready
+        print("Waiting for server to start...")
+        if wait_for_server(args.port):
+            print("✓ Web server started successfully")
+        else:
+            print("\n⚠️  Server may still be starting...")
+
+        # Step 5: Open browser (unless --no-browser flag is set)
+        url = f"http://127.0.0.1:{args.port}"
+        if not args.no_browser:
+            print(f"\nOpening browser to: {url}")
+            webbrowser.open(url)
+        else:
+            print(f"\nServer ready at: {url}")
+
+        print("\n" + "=" * 50)
+        print("Web interface is running!")
+        print(f"URL: {url}")
+        print("Press Ctrl+C to stop the server")
+        print("=" * 50 + "\n")
+
+        # Keep main thread alive
+        try:
+            server_thread.join()
+        except KeyboardInterrupt:
+            print("\n\nShutting down web server...")
+            cleanup()
+            print("Web server stopped.")
+
+        return 0
+
+    except ConfigurationError as e:
+        print(f"\n❌ Configuration Error: {e}")
+        return 1
+    except AuthenticationError as e:
+        print(f"\n❌ Authentication Error: {e}")
+        return 1
+    except Exception as e:
+        print(f"\n❌ Unexpected Error: {e}")
+        import traceback
+        traceback.print_exc()
+        return 1
+
+
 # Now import and run the application
 if __name__ == "__main__":
     try:
         # Parse command line arguments
         args = parse_arguments()
-        
+
         # Set up event loop policy for Windows
         if sys.platform == "win32":
             asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
-
-        # Run async initialization
-        exit_code = asyncio.run(initialize_and_run(args))
+
+        # Choose mode based on --web flag
+        if args.web:
+            # Web interface mode
+            exit_code = asyncio.run(initialize_and_run_web(args))
+        else:
+            # Desktop GUI mode (tkinter)
+            exit_code = asyncio.run(initialize_and_run(args))
+
         sys.exit(exit_code)
-        
+
     except KeyboardInterrupt:
         print("\nApplication interrupted by user")
         sys.exit(0)