...
+
+// ❌ WRONG: Using id or no identifier
+
+
+```
+
+### Naming Patterns
+
+| Component Type | Pattern | Example |
+|----------------|---------|---------|
+| Page container | `{page}-page` | `settings-page` |
+| Tab buttons | `tab-{tabId}` | `tab-api-keys` |
+| Wizard steps | `{wizard}-step-{stepId}` | `chronicle-step-llm` |
+| Form fields | `{context}-field-{name}` | `quickstart-field-openai-key` |
+| Secret inputs | `secret-input-{id}` | `secret-input-openai-key` |
+| Setting fields | `setting-field-{id}` | `setting-field-model-name` |
+| Buttons/Actions | `{context}-{action}` | `quickstart-refresh-status` |
+
+### Example POM Usage
+
+```typescript
+import { SettingsPage, WizardPage } from './pom'
+
+test('configure API keys', async ({ page }) => {
+ const settings = new SettingsPage(page)
+ await settings.goto()
+ await settings.waitForLoad()
+
+ await settings.goToApiKeysTab()
+ await settings.fillSecret('openai_api_key', 'sk-test-key')
+ await settings.expectApiKeyConfigured('openai_api_key')
+})
+
+test('complete quickstart wizard', async ({ page }) => {
+ const wizard = new WizardPage(page)
+ await wizard.startQuickstart()
+
+ await wizard.fillApiKey('openai_api_key', 'sk-test-key')
+ await wizard.next()
+ await wizard.waitForSuccess()
+})
+```
-## Next Steps (Immediate Actions)
+## Test Creation Workflow
-1. ✅ Create backend test infrastructure (Phase 1)
-2. ✅ Write initial backend tests for auth and services
-3. ✅ Set up Playwright configuration
-4. ✅ Create initial frontend E2E tests
-5. ✅ Set up CI/CD pipelines
-6. 📋 Generate code quality report
+When implementing a new feature:
-## Success Metrics
+1. **Specification Phase**: Use `/spec-agent` to create feature specification
+2. **Test Case Design**: Use `/qa-agent` to generate test scenarios
+3. **Test Automation**: Use `/automation-agent` to generate test code at appropriate levels
-- ✅ All critical paths tested
-- ✅ CI/CD pipeline running tests on every PR
-- ✅ Test execution time < 5 minutes
-- ✅ 80%+ code coverage on critical services
-- ✅ Zero Robot Framework dependencies (migrated to pytest/Playwright)
-- ✅ Clear testing documentation for contributors
+The automation agent will:
+- Determine correct test level (unit/integration/e2e)
+- Choose appropriate framework (pytest/Robot/Playwright)
+- Apply correct markers (`@pytest.mark.no_secrets` vs `@pytest.mark.requires_secrets`)
+- Add `data-testid` attributes to frontend code
+- Generate POM methods for E2E tests
-## Conclusion
+---
-This strategy prioritizes:
-1. **Modern tooling** (pytest, Playwright over Robot Framework)
-2. **Developer experience** (co-located tests, familiar tools)
-3. **Fast feedback** (quick test execution)
-4. **Maintainability** (clear structure, good practices)
+## References
-The Robot Framework tests in `tests_old/` should be treated as **reference material** during migration, then archived. They represent a valuable investment but are not the right tool for ongoing development.
+- [Test Pyramid Strategy Guide 2025](https://fullscale.io/blog/modern-test-pyramid-guide/)
+- [Playwright vs Robot Framework Comparison](https://www.browserstack.com/guide/playwright-vs-robot-framework)
+- [Robot Framework Browser Library Documentation](https://github.com/MarketSquare/robotframework-browser)
diff --git a/robot_tests/api/memory_feedback.robot b/robot_tests/api/memory_feedback.robot
new file mode 100644
index 00000000..637f4ee4
--- /dev/null
+++ b/robot_tests/api/memory_feedback.robot
@@ -0,0 +1,121 @@
+*** Settings ***
+Documentation Memory Feedback API Tests
+... Generated from: specs/features/memory-feedback.testcases.md
+...
+... Test Cases Covered:
+... - TC-MF-012: Submit True Feedback via API
+... - TC-MF-013: Submit Correction via API
+... - TC-MF-014: API Validation - Missing Corrected Text
+... - TC-MF-015: API Validation - Invalid Memory ID
+
+Library RequestsLibrary
+Library Collections
+Library String
+
+Suite Setup Create Session api ${BACKEND_URL} verify=False
+Suite Teardown Delete All Sessions
+
+*** Variables ***
+${BACKEND_URL} http://localhost:8000
+${VALID_MEMORY_ID} 507f1f77bcf86cd799439011
+${INVALID_MEMORY_ID} 000000000000000000000000
+${AUTH_TOKEN} Bearer test_jwt_token_here
+
+
+*** Test Cases ***
+TC-MF-012: Submit True Feedback via API
+ [Documentation] Test POST /api/memories/{id}/feedback with feedback_type="true"
+ [Tags] api no_secrets critical
+
+ # Given: Valid authentication and memory ID
+ ${headers}= Create Dictionary
+ ... Authorization=${AUTH_TOKEN}
+ ... Content-Type=application/json
+
+ ${payload}= Create Dictionary
+ ... feedback_type=true
+
+ # When: Submit feedback
+ ${response}= POST On Session api
+ ... /api/memories/${VALID_MEMORY_ID}/feedback
+ ... json=${payload}
+ ... headers=${headers}
+ ... expected_status=200
+
+ # Then: Verify response
+ Should Be Equal As Strings ${response.status_code} 200
+ Dictionary Should Contain Key ${response.json()} memory_id
+ Dictionary Should Contain Key ${response.json()} status
+ Dictionary Should Contain Key ${response.json()} feedback_count
+ Should Be Equal ${response.json()}[feedback_count][true] ${1}
+
+
+TC-MF-013: Submit Correction via API
+ [Documentation] Test POST /api/memories/{id}/feedback with correction
+ [Tags] api no_secrets critical
+
+ # Given: Correction payload
+ ${headers}= Create Dictionary
+ ... Authorization=${AUTH_TOKEN}
+ ... Content-Type=application/json
+
+ ${payload}= Create Dictionary
+ ... feedback_type=correction
+ ... corrected_text=The meeting was on Tuesday, not Monday
+
+ # When: Submit correction
+ ${response}= POST On Session api
+ ... /api/memories/${VALID_MEMORY_ID}/feedback
+ ... json=${payload}
+ ... headers=${headers}
+ ... expected_status=200
+
+ # Then: Verify response shows corrected status
+ Should Be Equal ${response.json()}[status] corrected
+ Dictionary Should Contain Key ${response.json()} feedback_count
+
+
+TC-MF-014: API Validation - Missing Corrected Text
+ [Documentation] Verify API returns 400 when corrected_text is missing
+ [Tags] api no_secrets negative
+
+ # Given: Correction without corrected_text
+ ${headers}= Create Dictionary
+ ... Authorization=${AUTH_TOKEN}
+ ... Content-Type=application/json
+
+ ${payload}= Create Dictionary
+ ... feedback_type=correction
+
+ # When/Then: Request should fail with 400
+ ${response}= POST On Session api
+ ... /api/memories/${VALID_MEMORY_ID}/feedback
+ ... json=${payload}
+ ... headers=${headers}
+ ... expected_status=400
+
+ # Verify error message
+ Should Contain ${response.json()}[error] corrected_text is required
+
+
+TC-MF-015: API Validation - Invalid Memory ID
+ [Documentation] Verify API returns 404 for non-existent memory
+ [Tags] api no_secrets negative
+
+ # Given: Non-existent memory ID
+ ${headers}= Create Dictionary
+ ... Authorization=${AUTH_TOKEN}
+ ... Content-Type=application/json
+
+ ${payload}= Create Dictionary
+ ... feedback_type=true
+
+ # When/Then: Request should fail with 404
+ ${response}= POST On Session api
+ ... /api/memories/${INVALID_MEMORY_ID}/feedback
+ ... json=${payload}
+ ... headers=${headers}
+ ... expected_status=404
+
+ # Verify error message
+ Should Contain ${response.json()}[error] Memory not found
diff --git a/scripts/verify-frontend-testids.sh b/scripts/verify-frontend-testids.sh
new file mode 100755
index 00000000..413836b7
--- /dev/null
+++ b/scripts/verify-frontend-testids.sh
@@ -0,0 +1,113 @@
+#!/bin/bash
+#
+# Verify Frontend Test IDs
+#
+# This script checks that all interactive frontend elements have data-testid attributes.
+# Run this after making frontend changes to ensure test automation requirements are met.
+#
+# Usage:
+# ./scripts/verify-frontend-testids.sh [file1.tsx file2.tsx ...]
+#
+# If no files are specified, checks all modified .tsx files in git.
+
+set -e
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+# Function to check a single file for interactive elements without data-testid
+check_file() {
+ local file=$1
+ local issues=0
+
+ echo "Checking $file..."
+
+ # Define patterns for interactive elements that MUST have data-testid
+ local patterns=(
+ 'Test & "quotes"
` + +**Expected Results** +- All malicious scripts are escaped or stripped +- Safe HTML entities are properly encoded +- Plain text is preserved + +--- + +### TC-MF-004: Calculate Memory Status - Verified + +**Type**: Unit +**Priority**: High +**Requires Secrets**: No + +**Description** +Verify status calculation when memory has >= 3 true feedbacks + +**Test Data** +```json +{ + "feedback_summary": {"true": 3, "false": 0, "correction": 0} +} +``` + +**Expected Results** +- Status = "verified" + +--- + +### TC-MF-005: Calculate Memory Status - Disputed + +**Type**: Unit +**Priority**: High +**Requires Secrets**: No + +**Description** +Verify status calculation when memory has >= 2 false feedbacks + +**Test Data** +```json +{ + "feedback_summary": {"true": 1, "false": 2, "correction": 0} +} +``` + +**Expected Results** +- Status = "disputed" + +--- + +### TC-MF-006: Calculate Memory Status - Corrected + +**Type**: Unit +**Priority**: Medium +**Requires Secrets**: No + +**Description** +Verify status calculation when memory has corrections + +**Test Data** +```json +{ + "feedback_summary": {"true": 1, "false": 0, "correction": 1} +} +``` + +**Expected Results** +- Status = "corrected" + +--- + +## Integration Tests (5 tests - 3 require secrets) + +### TC-MF-007: Store Feedback in Database + +**Type**: Integration +**Priority**: Critical +**Requires Secrets**: No + +**Description** +Verify feedback is correctly stored in memory_feedback collection + +**Preconditions** +- MongoDB test database running +- Test user authenticated +- Test memory exists + +**Test Steps** +1. Submit feedback with type="true" +2. Query memory_feedback collection +3. Verify document created with correct fields + +**Expected Results** +- Document exists with: + - memory_id (correct ObjectId) + - user_id (correct ObjectId) + - feedback_type = "true" + - created_at (timestamp) + - synced_to_server = false + - sync_attempts = 0 + +--- + +### TC-MF-008: Update Memory Feedback Summary + +**Type**: Integration +**Priority**: Critical +**Requires Secrets**: No + +**Description** +Verify memory document feedback_summary is updated when feedback is submitted + +**Preconditions** +- Test memory exists with feedback_summary = {true: 0, false: 0, correction: 0} + +**Test Steps** +1. Submit "true" feedback +2. Query memory document +3. Submit another "true" feedback +4. Query memory document again + +**Expected Results** +- After step 2: feedback_summary.true = 1 +- After step 4: feedback_summary.true = 2 + +--- + +### TC-MF-009: Memory Server Sync Success + +**Type**: Integration +**Priority**: High +**Requires Secrets**: **Yes** (MEMORY_SERVER_API_KEY) + +**Description** +Verify successful sync to memory server + +**Preconditions** +- Memory server API accessible +- Valid MEMORY_SERVER_API_KEY configured + +**Test Steps** +1. Submit feedback +2. Background job processes sync queue +3. Verify HTTP request sent to memory server +4. Verify feedback document updated + +**Expected Results** +- HTTP POST sent to memory server with feedback data +- Response 200 OK received +- feedback.synced_to_server = true +- feedback.sync_attempts = 1 + +--- + +### TC-MF-010: Memory Server Sync Failure with Retry + +**Type**: Integration +**Priority**: High +**Requires Secrets**: **Yes** (MEMORY_SERVER_API_KEY) + +**Description** +Verify retry mechanism when memory server is unavailable + +**Preconditions** +- Memory server mock returns 503 error + +**Test Steps** +1. Submit feedback +2. First sync attempt (mock returns 503) +3. Verify retry scheduled with exponential backoff +4. Second sync attempt succeeds (mock returns 200) + +**Expected Results** +- After step 2: + - synced_to_server = false + - sync_attempts = 1 + - last_sync_error contains error message +- After step 4: + - synced_to_server = true + - sync_attempts = 2 + +--- + +### TC-MF-011: Sync Queue Persistence + +**Type**: Integration +**Priority**: Medium +**Requires Secrets**: **Yes** (requires mock memory server) + +**Description** +Verify sync queue survives service restart + +**Test Steps** +1. Submit 5 feedbacks +2. Mock memory server as unavailable +3. Verify 5 items in sync queue +4. Restart backend service +5. Verify 5 items still in queue +6. Bring memory server back online +7. Verify all 5 synced + +**Expected Results** +- Sync queue persisted across restart +- All feedbacks eventually synced when server available + +--- + +## API Tests (5 tests - 1 requires secrets) + +### TC-MF-012: Submit True Feedback via API + +**Type**: API +**Priority**: Critical +**Requires Secrets**: No + +**Description** +Test POST /api/memories/{id}/feedback with feedback_type="true" + +**Preconditions** +- User authenticated (valid JWT) +- Memory exists in database + +**Test Steps** +1. POST /api/memories/{memory_id}/feedback + ```json + { + "feedback_type": "true" + } + ``` + +**Expected Results** +- Status: 200 OK +- Response body: + ```json + { + "memory_id": "{id}", + "status": "unverified", + "feedback_count": {"true": 1, "false": 0, "correction": 0}, + "updated_at": "{timestamp}", + "synced_to_server": false + } + ``` + +--- + +### TC-MF-013: Submit Correction via API + +**Type**: API +**Priority**: Critical +**Requires Secrets**: No + +**Description** +Test POST /api/memories/{id}/feedback with correction + +**Test Steps** +1. POST /api/memories/{memory_id}/feedback + ```json + { + "feedback_type": "correction", + "corrected_text": "The meeting was on Tuesday, not Monday" + } + ``` + +**Expected Results** +- Status: 200 OK +- Response contains memory_id, status="corrected" +- Feedback stored with corrected_text + +--- + +### TC-MF-014: API Validation - Missing Corrected Text + +**Type**: API +**Priority**: High +**Requires Secrets**: No + +**Description** +Verify API returns 400 when corrected_text is missing for correction type + +**Test Steps** +1. POST /api/memories/{memory_id}/feedback + ```json + { + "feedback_type": "correction" + } + ``` + +**Expected Results** +- Status: 400 Bad Request +- Error message: "corrected_text is required when feedback_type is 'correction'" + +--- + +### TC-MF-015: API Validation - Invalid Memory ID + +**Type**: API +**Priority**: High +**Requires Secrets**: No + +**Description** +Verify API returns 404 for non-existent memory + +**Test Steps** +1. POST /api/memories/000000000000000000000000/feedback + ```json + { + "feedback_type": "true" + } + ``` + +**Expected Results** +- Status: 404 Not Found +- Error message: "Memory not found" + +--- + +### TC-MF-016: API Rate Limiting + +**Type**: API +**Priority**: Medium +**Requires Secrets**: No + +**Description** +Verify rate limiting (max 100 feedbacks per user per hour) + +**Test Steps** +1. Submit 100 feedback requests rapidly +2. Submit 101st request + +**Expected Results** +- Requests 1-100: Success (200 OK) +- Request 101: 429 Too Many Requests +- Error includes retry-after information + +--- + +## E2E Tests (3 tests - all no_secrets) + +### TC-MF-017: User Marks Memory as True + +**Type**: E2E +**Priority**: Critical +**Requires Secrets**: No + +**Description** +Test complete workflow for marking memory as true via UI + +**Preconditions** +- User logged in +- Memory visible in UI + +**Test Steps** +1. Navigate to page with memory display +2. Hover over memory to reveal feedback buttons +3. Click "True" button (data-testid="memory-feedback-true") +4. Verify success animation/message appears +5. Verify button state changes to "active/selected" + +**Expected Results** +- Button click triggers API call +- Success feedback shown to user +- Button visual state updates +- Memory status may update if threshold reached + +--- + +### TC-MF-018: User Provides Correction + +**Type**: E2E +**Priority**: Critical +**Requires Secrets**: No + +**Description** +Test complete correction workflow via UI + +**Preconditions** +- User logged in +- Memory visible + +**Test Steps** +1. Click "Almost" button (data-testid="memory-feedback-almost") +2. Verify correction modal opens (data-testid="correction-modal") +3. Verify original text displayed +4. Enter corrected text in field (data-testid="correction-text-field") +5. Click Submit (data-testid="correction-submit") +6. Verify success message +7. Verify modal closes + +**Expected Results** +- Modal opens with original text visible +- User can type correction +- Submit sends API request +- Success feedback shown +- Modal auto-closes on success + +--- + +### TC-MF-019: Cancel Correction Modal + +**Type**: E2E +**Priority**: Medium +**Requires Secrets**: No + +**Description** +Verify user can cancel correction without submitting + +**Test Steps** +1. Click "Almost" button +2. Modal opens +3. Type some text in correction field +4. Click Cancel (data-testid="correction-cancel") + +**Expected Results** +- Modal closes +- No API request sent +- Memory state unchanged +- User can retry if desired + +--- + +## Test Coverage Matrix + +| Requirement | Test Cases | Coverage | +|-------------|-----------|----------| +| FR-001: Feedback UI Controls | TC-MF-017, TC-MF-018, TC-MF-019 | ✅ Happy Path, ⚠️ Edge Cases (cancel) | +| FR-002: Correction Input | TC-MF-018, TC-MF-019 | ✅ Happy Path, ⚠️ Edge Cases | +| FR-003: Feedback API | TC-MF-012, TC-MF-013, TC-MF-014, TC-MF-015, TC-MF-016 | ✅ Happy Path, ⚠️ Edge Cases, ❌ Negative | +| FR-004: Memory Server Sync | TC-MF-009, TC-MF-010, TC-MF-011 | ✅ Happy Path, ❌ Failure scenarios | +| FR-005: Feedback History | (Not yet covered - future enhancement) | ⚠️ Partial | +| NFR-001: Performance | (Implicit in API response time requirements) | ⚠️ Manual verification | +| NFR-002: Reliability | TC-MF-010, TC-MF-011 | ✅ Retry logic, queue persistence | +| NFR-003: Security | TC-MF-003, TC-MF-016 | ✅ XSS prevention, ❌ Rate limiting | + +--- + +## Secret Requirements Summary + +**Tests Requiring Secrets (4 tests)**: +- TC-MF-009: Memory Server Sync Success +- TC-MF-010: Memory Server Sync Failure with Retry +- TC-MF-011: Sync Queue Persistence + +These tests require `MEMORY_SERVER_API_KEY` environment variable. + +**Tests Without Secrets (15 tests)**: +- All unit tests (6) +- Most integration tests (2 of 5) +- All API tests (5) +- All E2E tests (3) + +**CI/CD Strategy**: +- 79% of tests (15/19) can run on every PR without secrets +- Only 21% (4/19) require manual trigger with secrets + +--- + +## Review Checklist + +Before approving for automation: + +- [x] All functional requirements have test cases +- [x] Happy path scenarios covered +- [x] Edge cases identified (empty input, validation, cancellation) +- [x] Negative tests included (invalid data, missing fields, non-existent resources) +- [x] Test data is realistic and sufficient +- [x] Dependencies are documented +- [x] Security considerations addressed (XSS, rate limiting) +- [x] Secret requirements clearly marked + +--- + +## Approval + +- [ ] QA Lead Approval +- [ ] Product Owner Approval +- [ ] Ready for Automation + +**Approved By**: _______________ +**Date**: _______________ diff --git a/ushadow/backend/pyproject.toml b/ushadow/backend/pyproject.toml index 3c6f7be4..977a132d 100644 --- a/ushadow/backend/pyproject.toml +++ b/ushadow/backend/pyproject.toml @@ -90,3 +90,17 @@ ignore = ["E501"] [tool.pytest.ini_options] asyncio_mode = "auto" testpaths = ["tests"] + +# Custom markers for test categorization +markers = [ + "unit: Unit tests that don't require external dependencies", + "integration: Integration tests that require services (DB, Redis, etc.)", + "e2e: End-to-end tests that test full workflows", + "requires_secrets: Tests that require API keys or secrets (skipped in CI)", + "no_secrets: Tests that run without any secrets (safe for PR checks)", + "requires_backend: Tests that need backend services running", + "requires_frontend: Tests that need frontend running", +] + +# Default: run only tests without secrets in CI +addopts = "-v --strict-markers" diff --git a/ushadow/backend/tests/integration/test_memory_server_sync.py b/ushadow/backend/tests/integration/test_memory_server_sync.py new file mode 100644 index 00000000..2c1acc46 --- /dev/null +++ b/ushadow/backend/tests/integration/test_memory_server_sync.py @@ -0,0 +1,193 @@ +""" +Integration tests for memory server synchronization. + +Generated by automation-agent from test cases in: +specs/features/memory-feedback.testcases.md + +Test Cases Covered: +- TC-MF-009: Memory Server Sync Success (REQUIRES SECRETS) +- TC-MF-010: Memory Server Sync Failure with Retry (REQUIRES SECRETS) +- TC-MF-011: Sync Queue Persistence (REQUIRES SECRETS) +""" + +import os +from unittest.mock import AsyncMock, patch + +import pytest + + +# TC-MF-009: Memory Server Sync Success +@pytest.mark.integration +@pytest.mark.requires_secrets +async def test_memory_server_sync_success(): + """ + Test Case: TC-MF-009 + + Steps: + 1. Submit feedback + 2. Background job processes sync queue + 3. Verify HTTP request sent to memory server + 4. Verify feedback document updated + + Expected: + - HTTP POST sent to memory server + - Response 200 OK + - feedback.synced_to_server = true + """ + from src.services.memory_feedback import MemoryFeedbackService + from src.services.memory_server_sync import MemoryServerSyncService + + # Skip if no API key configured + api_key = os.getenv("MEMORY_SERVER_API_KEY") + if not api_key: + pytest.skip("MEMORY_SERVER_API_KEY not configured") + + # Arrange + feedback_service = MemoryFeedbackService() + sync_service = MemoryServerSyncService() + + # Create test feedback + feedback_id = await feedback_service.submit_feedback( + memory_id="507f1f77bcf86cd799439011", + user_id="507f1f77bcf86cd799439012", + feedback_type="true", + ) + + # Act + result = await sync_service.sync_feedback(feedback_id) + + # Assert + assert result.success is True + assert result.response_status == 200 + + # Verify feedback updated + feedback = await feedback_service.get_feedback(feedback_id) + assert feedback.synced_to_server is True + assert feedback.sync_attempts == 1 + assert feedback.last_sync_error is None + + +# TC-MF-010: Memory Server Sync Failure with Retry +@pytest.mark.integration +@pytest.mark.requires_secrets +async def test_memory_server_sync_retry_on_failure(): + """ + Test Case: TC-MF-010 + + Steps: + 1. Submit feedback + 2. First sync attempt fails (503) + 3. Verify retry scheduled + 4. Second attempt succeeds + + Expected: + - After failure: synced_to_server=false, sync_attempts=1, error logged + - After success: synced_to_server=true, sync_attempts=2 + """ + from src.services.memory_feedback import MemoryFeedbackService + from src.services.memory_server_sync import MemoryServerSyncService + + # Skip if no API key + api_key = os.getenv("MEMORY_SERVER_API_KEY") + if not api_key: + pytest.skip("MEMORY_SERVER_API_KEY not configured") + + # Arrange + feedback_service = MemoryFeedbackService() + sync_service = MemoryServerSyncService() + + feedback_id = await feedback_service.submit_feedback( + memory_id="507f1f77bcf86cd799439011", + user_id="507f1f77bcf86cd799439012", + feedback_type="false", + ) + + # Act - First attempt (mock 503 error) + with patch.object(sync_service, "_make_request") as mock_request: + mock_request.return_value = AsyncMock(status_code=503) + result_1 = await sync_service.sync_feedback(feedback_id) + + # Assert - After first failure + assert result_1.success is False + feedback = await feedback_service.get_feedback(feedback_id) + assert feedback.synced_to_server is False + assert feedback.sync_attempts == 1 + assert feedback.last_sync_error is not None + + # Act - Second attempt (succeeds) + result_2 = await sync_service.sync_feedback(feedback_id) + + # Assert - After success + assert result_2.success is True + feedback = await feedback_service.get_feedback(feedback_id) + assert feedback.synced_to_server is True + assert feedback.sync_attempts == 2 + + +# TC-MF-011: Sync Queue Persistence +@pytest.mark.integration +@pytest.mark.requires_secrets +@pytest.mark.slow # This test involves service restart simulation +async def test_sync_queue_persistence_across_restart(): + """ + Test Case: TC-MF-011 + + Steps: + 1. Submit 5 feedbacks + 2. Mock memory server as unavailable + 3. Verify 5 items in sync queue + 4. Simulate service restart + 5. Verify 5 items still in queue + 6. Bring server back online + 7. Verify all 5 synced + + Expected: Sync queue persists across restart + """ + from src.services.memory_feedback import MemoryFeedbackService + from src.services.memory_server_sync import MemoryServerSyncService + + # Skip if no API key + api_key = os.getenv("MEMORY_SERVER_API_KEY") + if not api_key: + pytest.skip("MEMORY_SERVER_API_KEY not configured") + + # Arrange + feedback_service = MemoryFeedbackService() + sync_service = MemoryServerSyncService() + + # Submit 5 feedbacks + feedback_ids = [] + for i in range(5): + fid = await feedback_service.submit_feedback( + memory_id="507f1f77bcf86cd799439011", + user_id="507f1f77bcf86cd799439012", + feedback_type="true", + ) + feedback_ids.append(fid) + + # Mock server unavailable + with patch.object(sync_service, "_make_request") as mock_request: + mock_request.return_value = AsyncMock(status_code=503) + for fid in feedback_ids: + await sync_service.sync_feedback(fid) + + # Assert - 5 items in queue (not synced) + queue_size = await sync_service.get_queue_size() + assert queue_size == 5 + + # Simulate service restart (close and reopen connection) + await sync_service.close() + sync_service = MemoryServerSyncService() # Fresh instance + + # Assert - Queue still has 5 items + queue_size = await sync_service.get_queue_size() + assert queue_size == 5 + + # Bring server back online (real requests now) + for fid in feedback_ids: + result = await sync_service.sync_feedback(fid) + assert result.success is True + + # Assert - All synced + queue_size = await sync_service.get_queue_size() + assert queue_size == 0 diff --git a/ushadow/backend/tests/test_memory_feedback_validation.py b/ushadow/backend/tests/test_memory_feedback_validation.py new file mode 100644 index 00000000..e2f7a283 --- /dev/null +++ b/ushadow/backend/tests/test_memory_feedback_validation.py @@ -0,0 +1,172 @@ +""" +Test module for memory feedback validation logic. + +Generated by automation-agent from test cases in: +specs/features/memory-feedback.testcases.md + +Test Cases Covered: +- TC-MF-001: Validate Feedback Type +- TC-MF-002: Validate Corrected Text Length +- TC-MF-003: Sanitize Corrected Text for XSS +- TC-MF-004: Calculate Memory Status - Verified +- TC-MF-005: Calculate Memory Status - Disputed +- TC-MF-006: Calculate Memory Status - Corrected +""" + +import pytest + + +# TC-MF-001: Validate Feedback Type +@pytest.mark.unit +@pytest.mark.no_secrets +def test_validate_feedback_type_valid_values(): + """ + Test Case: TC-MF-001 (Valid feedback types) + + Steps: + 1. Validate feedback_type="true" + 2. Validate feedback_type="false" + 3. Validate feedback_type="correction" + + Expected: All validations pass + """ + from src.validators.memory_feedback import validate_feedback_type + + # Arrange & Act & Assert + assert validate_feedback_type("true") is True + assert validate_feedback_type("false") is True + assert validate_feedback_type("correction") is True + + +@pytest.mark.unit +@pytest.mark.no_secrets +def test_validate_feedback_type_invalid_values(): + """ + Test Case: TC-MF-001 (Invalid feedback types) + + Steps: + 1. Validate feedback_type="invalid" + 2. Validate feedback_type="" + 3. Validate feedback_type=None + + Expected: All validations fail with appropriate error + """ + from src.validators.memory_feedback import validate_feedback_type, ValidationError + + # Arrange + invalid_types = ["invalid", "", None, "TRUE", "False", 123] + + # Act & Assert + for invalid_type in invalid_types: + with pytest.raises(ValidationError, match="feedback_type must be"): + validate_feedback_type(invalid_type) + + +# TC-MF-002: Validate Corrected Text Length +@pytest.mark.unit +@pytest.mark.no_secrets +def test_validate_corrected_text_length(): + """ + Test Case: TC-MF-002 + + Steps: + 1. Validate empty string + 2. Validate 1 character + 3. Validate 2000 characters + 4. Validate 2001 characters + + Expected: Only 1-2000 chars pass + """ + from src.validators.memory_feedback import ( + validate_corrected_text, + ValidationError, + ) + + # Empty/None should fail + with pytest.raises(ValidationError, match="corrected_text is required"): + validate_corrected_text("") + + with pytest.raises(ValidationError, match="corrected_text is required"): + validate_corrected_text(None) + + # 1 char should pass + assert validate_corrected_text("a") is True + + # 2000 chars should pass + assert validate_corrected_text("x" * 2000) is True + + # 2001 chars should fail + with pytest.raises(ValidationError, match="exceeds maximum length"): + validate_corrected_text("x" * 2001) + + +# TC-MF-003: Sanitize Corrected Text for XSS +@pytest.mark.unit +@pytest.mark.no_secrets +def test_sanitize_corrected_text_xss_prevention(): + """ + Test Case: TC-MF-003 + + Steps: + 1. Input malicious script tags + 2. Input image with onerror + 3. Input javascript: protocol + 4. Input normal text with HTML entities + + Expected: All malicious code escaped/stripped, safe text preserved + """ + from src.utils.sanitize import sanitize_html + + # Malicious inputs + assert "") + assert "onerror" not in sanitize_html("Test & "quotes"
') + assert "<" in result or "" not in result # Tags escaped or stripped + + # Plain text preserved + plain_text = "This is normal text without HTML" + assert sanitize_html(plain_text) == plain_text + + +# TC-MF-004, TC-MF-005, TC-MF-006: Calculate Memory Status +@pytest.mark.unit +@pytest.mark.no_secrets +@pytest.mark.parametrize( + "feedback_summary,expected_status", + [ + # TC-MF-004: Verified (>= 3 true) + ({"true": 3, "false": 0, "correction": 0}, "verified"), + ({"true": 5, "false": 1, "correction": 0}, "verified"), + # TC-MF-005: Disputed (>= 2 false) + ({"true": 1, "false": 2, "correction": 0}, "disputed"), + ({"true": 0, "false": 3, "correction": 0}, "disputed"), + # TC-MF-006: Corrected (has corrections) + ({"true": 1, "false": 0, "correction": 1}, "corrected"), + ({"true": 2, "false": 1, "correction": 2}, "corrected"), + # Edge case: Unverified (no significant feedback) + ({"true": 1, "false": 0, "correction": 0}, "unverified"), + ({"true": 2, "false": 1, "correction": 0}, "unverified"), + ], +) +def test_calculate_memory_status(feedback_summary, expected_status): + """ + Test Cases: TC-MF-004, TC-MF-005, TC-MF-006 + + Verifies memory status calculation based on feedback counts. + + Rules: + - >= 3 true feedbacks → "verified" + - >= 2 false feedbacks → "disputed" + - > 0 corrections → "corrected" + - Otherwise → "unverified" + """ + from src.models.memory import calculate_memory_status + + # Act + status = calculate_memory_status(feedback_summary) + + # Assert + assert status == expected_status diff --git a/ushadow/frontend/e2e/memory-feedback.spec.ts b/ushadow/frontend/e2e/memory-feedback.spec.ts new file mode 100644 index 00000000..0a2d7ffd --- /dev/null +++ b/ushadow/frontend/e2e/memory-feedback.spec.ts @@ -0,0 +1,145 @@ +/** + * E2E Tests: Memory Feedback + * + * Generated from: specs/features/memory-feedback.testcases.md + * + * Test Cases Covered: + * - TC-MF-017: User Marks Memory as True + * - TC-MF-018: User Provides Correction + * - TC-MF-019: Cancel Correction Modal + */ + +import { test, expect } from '@playwright/test' + +test.describe('Memory Feedback', () => { + test.beforeEach(async ({ page }) => { + // Setup: Login and navigate to memories page + await page.goto('/login') + await page.fill('[data-testid="email-field"]', 'test@example.com') + await page.fill('[data-testid="password-field"]', 'testpass123') + await page.click('[data-testid="login-button"]') + await page.waitForURL('/memories') + }) + + /** + * TC-MF-017: User Marks Memory as True + * + * Priority: Critical + * Requires Secrets: No + */ + test('TC-MF-017: user can mark memory as true', async ({ page }) => { + // Given: Memory is visible + const memory = page.locator('[data-testid="memory-item"]').first() + await expect(memory).toBeVisible() + + // When: User clicks True button + await memory.locator('[data-testid="memory-feedback-true"]').click() + + // Then: Success feedback shown + await expect(page.locator('[data-testid="feedback-success-toast"]')).toBeVisible() + await expect(page.locator('[data-testid="feedback-success-toast"]')).toContainText( + 'Feedback submitted' + ) + + // And: Button state updates + const trueButton = memory.locator('[data-testid="memory-feedback-true"]') + await expect(trueButton).toHaveClass(/active|selected/) + }) + + /** + * TC-MF-018: User Provides Correction + * + * Priority: Critical + * Requires Secrets: No + */ + test('TC-MF-018: user can provide correction', async ({ page }) => { + // Given: Memory is visible + const memory = page.locator('[data-testid="memory-item"]').first() + const originalText = await memory.locator('[data-testid="memory-text"]').textContent() + + // When: User clicks Almost button + await memory.locator('[data-testid="memory-feedback-almost"]').click() + + // Then: Correction modal opens + const modal = page.locator('[data-testid="correction-modal"]') + await expect(modal).toBeVisible() + + // And: Original text is displayed + const displayedOriginal = await modal.locator('[data-testid="original-text"]').textContent() + expect(displayedOriginal).toContain(originalText) + + // When: User enters correction + const correctionText = 'The meeting was on Tuesday, not Monday' + await modal.locator('[data-testid="correction-text-field"]').fill(correctionText) + + // And: User clicks Submit + await modal.locator('[data-testid="correction-submit"]').click() + + // Then: Success message shown + await expect(page.locator('[data-testid="feedback-success-toast"]')).toBeVisible() + + // And: Modal closes + await expect(modal).not.toBeVisible() + + // And: Memory status may update + // (Status update depends on backend calculation, just verify no error) + await expect(page.locator('[data-testid="error-toast"]')).not.toBeVisible() + }) + + /** + * TC-MF-019: Cancel Correction Modal + * + * Priority: Medium + * Requires Secrets: No + */ + test('TC-MF-019: user can cancel correction modal', async ({ page }) => { + // Given: Memory is visible + const memory = page.locator('[data-testid="memory-item"]').first() + + // When: User clicks Almost button + await memory.locator('[data-testid="memory-feedback-almost"]').click() + + // Then: Modal opens + const modal = page.locator('[data-testid="correction-modal"]') + await expect(modal).toBeVisible() + + // When: User types some text + await modal + .locator('[data-testid="correction-text-field"]') + .fill('Some partial correction') + + // And: User clicks Cancel + await modal.locator('[data-testid="correction-cancel"]').click() + + // Then: Modal closes + await expect(modal).not.toBeVisible() + + // And: No API request was made (verify by checking no success toast) + await expect(page.locator('[data-testid="feedback-success-toast"]')).not.toBeVisible() + + // And: Memory state unchanged (Almost button still clickable) + await expect(memory.locator('[data-testid="memory-feedback-almost"]')).toBeVisible() + }) + + /** + * Additional test: Verify data-testid attributes exist + */ + test('all required test IDs are present', async ({ page }) => { + const memory = page.locator('[data-testid="memory-item"]').first() + await expect(memory).toBeVisible() + + // Verify all feedback buttons have test IDs + await expect(memory.locator('[data-testid="memory-feedback-true"]')).toBeVisible() + await expect(memory.locator('[data-testid="memory-feedback-false"]')).toBeVisible() + await expect(memory.locator('[data-testid="memory-feedback-almost"]')).toBeVisible() + + // Open modal to verify its test IDs + await memory.locator('[data-testid="memory-feedback-almost"]').click() + const modal = page.locator('[data-testid="correction-modal"]') + + await expect(modal.locator('[data-testid="original-text"]')).toBeVisible() + await expect(modal.locator('[data-testid="correction-text-field"]')).toBeVisible() + await expect(modal.locator('[data-testid="correction-submit"]')).toBeVisible() + await expect(modal.locator('[data-testid="correction-cancel"]')).toBeVisible() + }) +})