Feat: Databento integration improvements #10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…More robust code. Refactored and properly passing tests.
feat: Record Type of different DataBento scheme feat: handle Auth-Request/Response msgs test:feat: deserialize Heartbeat json msg, parse tcp response/request test:feat: LiveApiClient quick connection and debug process
Martin-Molinero
suggested changes
Jan 28, 2026
Member
Martin-Molinero
left a comment
Martin-Molinero
left a comment
There was a problem hiding this comment.
Thank you! Shared detailed review in slack, few requests and bug fixing required
Unify market data deserialization using a new DataConverter and MarketDataBase abstract class. Remove custom JsonSettings and legacy deserialization methods. Update all model classes to inherit from MarketDataBase, make LevelOneBookLevel prices nullable, and improve error handling for unknown/invalid JSON. Move event args to .Models.Events. Expand and update tests for new d eserialization logic and edge cases. These changes improve robustness, extensibility, and maintainability of DataBento data ingestion.
- Add ErrorResponse model for structured API error parsing - Refactor HistoricalAPIClient to auto-correct invalid date ranges - Expand unit tests for error scenarios and data bounds - Add tests for error response deserialization - Support additional OHLCV record types in DataConverter - Use snake-case JSON settings for consistent deserialization - Improve logging and handling of empty/malformed responses
Introduce DataSetSpecifications and PredefinedDataSets for richer dataset metadata, including delay info and documentation links. Refactor DataBentoSymbolMapper to map markets to DataSetSpecifications. Update provider logic to log delay warnings and extract DataSetID as needed. Add support for new error case DataStartAfterAvailableEnd and corresponding test. Remove obsolete methods and simplify subscription/history logic.
Replaced hardcoded schema strings with documented constants in HistoricalAPIClient for maintainability. Refactored GetRange to set record limits by schema, preventing timeouts on large requests. Added tests for 11-year minute and second data ranges. Updated all schema usages to use new constants.
Refactored DataBentoProvider to inherit from MappedSynchronizingHistoryProvider, removing custom map file handling and multi-request GetHistory logic. Cleaned up unused usings and simplified history request processing by leveraging the new base class for map file resolution and request splitting.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related PRs