Refacto/clean archi#6
Open
cyberbobjr wants to merge 13 commits into
Open
Conversation
fix: Correct datetime reference in get_earnings_dates function chore: Add settings.local.json for project configuration docs: Create CLAUDE.md for project overview and usage instructions chore: Initialize pyproject.toml for pytest configuration test: Add __init__.py to tests directory for package recognition test: Create conftest.py for shared fixtures in stockscreen tests test: Implement comprehensive unit tests for stockscreen.py functionality
…ckage - Added MIGRATION_PLAN.md detailing the migration strategy and phases. - Created initial package structure for stockscreen with necessary modules. - Migrated configuration and logging setup to stockscreen/config.py. - Established exception hierarchy in stockscreen/exceptions.py. - Developed Pydantic models for validation in stockscreen/models/schemas.py. - Implemented data persistence logic in stockscreen/store/data_store.py. - Set up Yahoo provider interface in stockscreen/providers/yahoo.py. - Created service modules for news, screener, and watchlist management in stockscreen/services/. - Added tests for configuration, exceptions, schemas, and data store functionality.
- Implement tests for the BaseSymbolFetcher interface, ensuring proper instantiation and compliance with abstract methods. - Create tests for Wikipedia-based symbol fetchers (SP500, Nasdaq100, CAC40, SBF120, DAX, FTSE100, AEX) to validate fetching logic and error handling. - Develop tests for SymbolService to verify caching behavior, refresh logic, and background scheduling. - Include various scenarios such as empty responses, network errors, and cache expiration to ensure robustness.
- Implement comprehensive unit tests for MarketDataFacade to validate identifier resolution, dividend strategy, technical data retrieval, and parallel calls. - Create tests for PalmaresService focusing on caching logic, sorting, filtering, and snapshot metadata. - Introduce tests for PalmaresStore to ensure proper loading, saving, and data integrity of snapshots. - Update server tests to include routes for PalmaresService and ensure correct parameter forwarding and error handling.
- Updated project overview in CLAUDE.md to reflect the addition of new tools: `refresh_symbols` and `get_palmares`. - Expanded the description of data sources in README.md, including details on `EuronextProvider` and `BoursoramaPalmaresScaper`. - Improved the documentation for various tools, including `get_stock_news`, `manage_watchlist`, `get_screening_result`, `refresh_symbols`, and `get_palmares`, with clearer parameter descriptions and example usage. - Updated server.py to enhance the functionality of `get_palmares`, ensuring it correctly handles cache refresh and filtering. - Added tests in test_server.py to verify the correct behavior of the `get_palmares` function, ensuring that parameters are forwarded to the service correctly.
- Replace requirements.txt with pyproject.toml dependency groups - Add build-system (hatchling) and hatch package config - Add uv.lock - Update README and CLAUDE.md with uv commands and uvx connection options
There was a problem hiding this comment.
Pull request overview
This PR refactors the project into a modular package architecture (providers/services/store/models), adds a unified market-data facade (Yahoo/Boursorama/Euronext) plus symbol-fetching and “palmarès dividendes” features, and introduces a comprehensive pytest suite and documentation to support the new structure.
Changes:
- Added new provider layer (Euronext resolver, MarketDataFacade, Wikipedia index symbol fetchers, Boursorama palmarès scraper) and new services (SymbolService, PalmaresService).
- Added file-based persistence for palmarès snapshots and expanded the data store / schema validation & JSON encoding.
- Added extensive unit + integration tests and migration/implementation documentation; moved dependencies to
pyproject.toml.
Reviewed changes
Copilot reviewed 51 out of 59 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_yahoo_provider.py | New unit tests for async YahooProvider wrapper and retry/executor behavior. |
| tests/test_watchlist_service.py | New tests for WatchlistService CRUD and validation. |
| tests/test_symbol_service.py | New tests for SymbolService caching, refresh, and background tick behavior. |
| tests/test_symbol_fetchers_wikipedia.py | New tests for Wikipedia-based index fetchers and suffix normalization. |
| tests/test_symbol_fetcher_base.py | New tests for BaseSymbolFetcher ABC + SymbolRecord validation. |
| tests/test_server.py | New/updated tests for FastMCP tool routing and service wiring (incl. palmarès). |
| tests/test_schemas.py | New tests for Pydantic schema validation and JSON encoding behavior. |
| tests/test_palmares_store.py | New tests for palmarès snapshot persistence (round-trip, corruption handling). |
| tests/test_palmares_service.py | New tests for palmarès caching, sorting, filtering, and refresh. |
| tests/test_news_service.py | New tests for news fetch/categorization and screening filters. |
| tests/test_market_data_facade.py | New tests for MarketDataFacade resolution/merge/fallback strategy. |
| tests/test_integration.py | New integration tests for screener + watchlists + provider interactions. |
| tests/test_exceptions.py | New tests for exception hierarchy. |
| tests/test_euronext_provider.py | New tests for EuronextProvider resolution + caching + executor usage. |
| tests/test_data_store.py | New tests for ScreenerDataStore and DefaultSymbols caching/filtering. |
| tests/test_config.py | New tests for config defaults and legacy data migration behavior. |
| tests/conftest.py | Shared fixtures + test-time data path isolation via env var. |
| tests/init.py | Marks tests as a package. |
| TASKS.md | Detailed TDD task checklist for the refactor and new features. |
| stockscreen/store/palmares_store.py | New PalmaresStore and PalmaresSnapshot persistence model. |
| stockscreen/store/data_store.py | ScreenerDataStore + DefaultSymbols persistence/cache implementation. |
| stockscreen/services/watchlist.py | New WatchlistService using Pydantic validation and store persistence. |
| stockscreen/services/symbol_service.py | New SymbolService for fetch/cache/refresh/background symbol lists. |
| stockscreen/services/palmares_service.py | New PalmaresService cache orchestration + filter/sort logic. |
| stockscreen/services/news.py | NewsService updated to use MarketDataFacade provider. |
| stockscreen/providers/yahoo.py | Async YahooProvider wrapper over yfinance with executor + retry. |
| stockscreen/providers/symbol_fetchers/wikipedia.py | Wikipedia-based index symbol fetchers using pd.read_html via executor. |
| stockscreen/providers/symbol_fetchers/registry.py | Registry + builder for enabled symbol fetcher sources. |
| stockscreen/providers/symbol_fetchers/base.py | BaseSymbolFetcher ABC + SymbolRecord dataclass + validation. |
| stockscreen/providers/symbol_fetchers/init.py | Public exports for symbol fetcher subsystem. |
| stockscreen/providers/facade.py | MarketDataFacade combining Yahoo/Boursorama/Euronext with merge strategy. |
| stockscreen/providers/euronext.py | EuronextProvider ISIN↔ticker resolver with caching and executor HTTP. |
| stockscreen/providers/boursorama_palmares.py | Boursorama palmarès multi-page scraper and PalmaresEntry model. |
| stockscreen/models/schemas.py | Pydantic models for watchlist/symbol validation + JSON encoder. |
| stockscreen/exceptions.py | StockscreenError / ValidationError / APIError exception types. |
| stockscreen/data/screening_results/top_dividend_yield.json | Added/updated sample screening result data. |
| stockscreen/data/screening_results/top_dividend_france.json | Added/updated sample screening result data. |
| stockscreen/data/screening_results/top_dividend_euronext_paris.json | Added/updated sample screening result data. |
| stockscreen/config.py | Centralized configuration + env overrides + logging + migration helpers. |
| stockscreen/init.py | Package-level exports aligned to modular architecture. |
| requirements.txt | Removed legacy requirements file in favor of pyproject.toml. |
| pyproject.toml | New project metadata/dependencies/scripts + pytest config. |
| MIGRATION_PLAN.md | Full migration plan from monolith to modular package (TDD-driven). |
| CLAUDE.md | Updated developer guidance reflecting new architecture and data flow. |
| .gitignore | Added ignores for logs, .mcp.json, and .claude/. |
| stockscreen/store/init.py | Package init file for store module. |
| stockscreen/services/init.py | Package init file for services module. |
| stockscreen/providers/init.py | Package init file for providers module. |
| stockscreen/models/init.py | Package init file for models module. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+101
to
+117
| return await self._yahoo.get_history(identifier, period=period) | ||
|
|
||
| async def get_news(self, identifier: str) -> list[dict]: | ||
| """Get recent news — delegates to Yahoo.""" | ||
| return await self._yahoo.get_news(identifier) | ||
|
|
||
| async def get_option_chain(self, identifier: str, expiry: str) -> Any: | ||
| """Get options chain — delegates to Yahoo.""" | ||
| return await self._yahoo.get_option_chain(identifier, expiry) | ||
|
|
||
| async def get_option_expirations(self, identifier: str) -> tuple: | ||
| """Get option expiration dates — delegates to Yahoo.""" | ||
| return await self._yahoo.get_option_expirations(identifier) | ||
|
|
||
| async def get_earnings_dates(self, identifier: str) -> dict: | ||
| """Get earnings dates — delegates to Yahoo.""" | ||
| return await self._yahoo.get_earnings_dates(identifier) |
Comment on lines
+42
to
+52
| news_item = { | ||
| "title": item.get("title"), | ||
| "publisher": item.get("publisher"), | ||
| "published_at": pub_time.isoformat(), | ||
| "type": item.get("type"), | ||
| "summary": item.get("summary"), | ||
| } | ||
|
|
||
| title_lower = news_item["title"].lower() | ||
| if any(t in title_lower for t in _MANAGEMENT_KEYWORDS): | ||
| news_data["management_changes"].append(news_item) |
Comment on lines
+30
to
+39
| def __init__( | ||
| self, | ||
| fetchers: list[BaseSymbolFetcher], | ||
| cache_dir: str, | ||
| refresh_interval_hours: float = 24.0, | ||
| ): | ||
| self._fetchers: dict[str, BaseSymbolFetcher] = {f.name: f for f in fetchers} | ||
| self._cache_dir = cache_dir | ||
| self._refresh_interval = refresh_interval_hours * 3600 | ||
| os.makedirs(cache_dir, exist_ok=True) |
Comment on lines
+87
to
+95
| async def _fetch_and_save(self) -> PalmaresSnapshot: | ||
| entries = await self._scraper.fetch_all() | ||
| snapshot = PalmaresSnapshot( | ||
| fetched_at=datetime.datetime.now().isoformat(), | ||
| page_count=0, | ||
| total_entries=len(entries), | ||
| entries=entries, | ||
| ) | ||
| self._store.save(snapshot) |
- Add STOCKSCREEN_TRANSPORT env var (stdio/sse/streamable-http) - Add STOCKSCREEN_HOST and STOCKSCREEN_PORT for HTTP transports - Dispatch to run_stdio_async/run_sse_async/run_streamable_http_async in main() - Add AGENTS.md with coding rules (French chat, English code, docs-update) - Add ARCHITECTURE.md as comprehensive architecture reference - Remove obsolete IMPLEMENTATION_PLAN.md, MIGRATION_PLAN.md, TASKS.md - Bump version to 2.1.0
…t, dup validation - facade: resolve ISIN via Euronext in all 5 delegated Yahoo methods - news: guard against None title/summary in categorization and screening - symbol_service: raise ValidationError on duplicate fetcher names - palmares: propagate real page_count from scraper to snapshot metadata - chore: remove unused APIError imports
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.