Skip to content

Implement protobuf#185

Open
gnzsnz wants to merge 48 commits intoib-api-reloaded:mainfrom
gnzsnz:protobuf
Open

Implement protobuf#185
gnzsnz wants to merge 48 commits intoib-api-reloaded:mainfrom
gnzsnz:protobuf

Conversation

@gnzsnz
Copy link
Contributor

@gnzsnz gnzsnz commented Oct 27, 2025

works with protobuf ONLY, requires 10.40 or greater
replaces wrapper._results with a reactive pipeline implemented in eventkit. This simplifies the solution by removing state management

simple test script

import logging
import ib_async


ib_async.util.startLoop()
logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s %(levelname)s %(name)s: %(message)s',
    filename='debug_connection.log',
    filemode='w')
ib = ib_async.IB()

ib.connect('localhost',7497,clientId=0)
print("connection completed")

aapl = ib_async.Stock("AAPL",'SMART','USD')
aapl_details = ib.reqContractDetails(aapl)
print(aapl)
print(aapl_details)

ib.disconnect()
print("disconnected.")

works with protobuf ONLY, requires 10.40 or greater
@gnzsnz
Copy link
Contributor Author

gnzsnz commented Oct 27, 2025

Still a long way to go, but I wanted to share a version that can connect, sync account related objects and perform some basic requests and disconnect.

Implementation status, with their cancel counterpart:

  • reqHistoricalData, working for one-off requests. Subscription pending
  • reqHistoricalData keepUpToDate=True
  • reqHistoricalTicks,
  • reqHeadTimeStamp
  • reqHistogramData
  • reqHistoricalSchedule
  • reqTickByTickData
  • reqMarketDataType
  • calculateImpliedVolatilityAsync
  • calculateOptionPriceAsync
  • placeOrder
  • cancelOrder - breaking change, new data type OrderCancel in parameter
  • whatIfOrder
  • reqGlobalCancel
  • tests: i have a few tests in place. i need to do some clean-up and i will push.
  • tests for data converters from/to protobuf
  • reqMktData,
  • cancelMktData
  • reqRealTimeBars,
  • reqScannerSubscription,
  • reqScannerDataAsync
  • cancelScannerSubscription
  • reqScannerParameters
  • reqPnL,
  • reqPnLSingle.
  • reqUserInfoAsync
  • reqTickers
  • reqSmartComponentsAsync
  • reqFundamentalDataAsync
  • request throttling
  • reqMktDepthExchanges
  • reqMktDepth
  • cancelMktDepth
  • getWshMetaDataAsync
  • reqWshMetaData
  • getWshEventDataAsync
  • reqWshEventData
  • reqNewsProvidersAsync
  • reqNewsArticleAsync
  • reqHistoricalNewsAsync
  • reqNewsBulletins
  • newsTicks
  • newsBulletins
  • requestFAAsync
  • replaceFA
  • reqWshMetaData

- Added logging for ticker data reception and processing.
- Introduced new methods in Ticker class for handling various tick data types (price, size, string, generic, computation).
- Updated Wrapper class to emit events for tick data using a bus system.
- Removed redundant tick type mappings and replaced them with more structured handling.
- Enhanced error handling for malformed tick data.
- Improved readability and maintainability of the code by organizing tick data processing logic.
@gnzsnz
Copy link
Contributor Author

gnzsnz commented Nov 10, 2025

added ticker protobuf support

- Introduced IneligibilityReason dataclass for better data structure.
- Updated ContractDetails to use a list of IneligibilityReason.
- historical schedule.
- Improved market data converters to handle edge cases.
- Cleaned up unused imports and methods across various modules.
gnzsnz added 17 commits December 3, 2025 19:41
- BiDict class to encapsulate wrapper state management
- reactive bus implemented: ticker_bus, response_bus, subscription_bus
- object logic encapsulated on the object itself, rather than in Wrapper class
- all subscriptions are implemented. realtimeBar, , keepUpToDate, scanners, PnL and PnLSingle
- placeOrder,cancelOrder, whatIfOrder
- Ticker and tickByTick
- Implement tests for subscription converters including ScannerParametersRequest, ScannerSubscriptionRequest, and PnL requests.
- Add tests for trade converters covering order creation, conditions, and execution details.
- Ensure comprehensive coverage for various order conditions and soft dollar tiers.
- Validate the conversion of protobuf messages to domain objects and vice versa.
- Most of methods are implemented, except for news related and market L2.
… improve data handling

- Added new tick types for ETF NAV including close, prior close, bid, ask, last, frozen last, high, and low.
- Updated Ticker dataclass to include new fields for ETF NAV data.
- Modified the __post_init__ method to initialize new ETF NAV fields.
- Improved the handling of ticker data updates to accommodate new tick types.
- Refactored ticker data processing methods for clarity and efficiency.

feat(util): Introduce quantize_decimals decorator for Decimal fields

- Added a decorator to quantize Decimal fields in dataclass objects to specified decimal places.
- Enhanced the parseIBDatetime function to correctly handle datetime strings.

refactor(wrapper): Streamline ticker and subscription handling

- Consolidated tick data handling methods to reduce redundancy.
- Updated method signatures to remove unnecessary parameters.
- Improved error handling for unknown request IDs in ticker delivery methods.

chore(tests): Update tests for market data and trade converters

- Refactored test cases to align with changes in the codebase.
- Ensured tests validate new fields and behaviors introduced in the Ticker class.
- Updated import paths for trade converter functions to reflect new module structure.
- Update tests for utility functions in tests_util.py
- remove quantize_decimals from trade_converters
feat(news): Add news bulletin handling and related protobuf converters
fix: fix comboLegs contract protobuf converters
feat: implement __repr__ with dataclassRepr for ContractDetails
fix: align ticker with main branch. EFP can't be implemented as there is no proto message.
- refactor: reorganize imports, use relative imports consistenly
- feat: news methods implemented, reqNewsProviders, reqHistoricalNews, reqNewsArticle, reqNewsBulletins, cancelNewsBulletins
=======

- Add support for position multi requests
- add support for soft dollar tiers
- add support for family codes
- Updated protobuf message handling in decoder.py to support bond contract details.
- Refactored createDeltaNeutralContract in contract_converters.py for improved clarity.
- Added positionMultiEvent to ib.py for enhanced event tracking.
- Adjusted wrapper.py to emit positionMultiEvent.
- Improved error handling in market_data_converters.py for tick data.
- Updated `conftest.py` to ignore type errors for mocked IB client methods.
- Enhanced `test_account_converters.py` with additional test cases for new protobuf converters.
- Created `test_base_converters.py` to cover base converter functionalities.
- Modified `test_contract_converters.py` to include new contract-related tests and improved type handling.
- Updated `test_contract_requests.py` to ensure proper mocking and type handling in contract requests.
- Expanded `test_historical_data_converters.py` with new tests for historical tick data and related converters.
- Enhanced `test_market_data_converters.py` with new tests for tick data handling.
- Introduced `test_news_converters.py` to validate news-related protobuf converters.
- Updated `test_subscription_converters.py` to improve test coverage for subscription-related converters.
- Refactored `test_trade_converter.py` to streamline imports and enhance test coverage for trade-related functionalities.
gnzsnz added 22 commits January 18, 2026 12:33
…by contract, and only create a new ticker if contract is not found.
Add _response_single and _response_multi to fix the issue and keep code DRY.
This commit solves the problem for all methods called during connection.

Problem explanation: by using `response_bus.filter` we create a problem. response
bus is a "long lived, common" event, it does not complete, so it's never set as
done. that means that `filter` is not calling `on_sorce_done` and itself is not
set to "done" state. however, `take(n)` and `takewhile(predicate)` work as
expected and disconnect from the source when the condition is meet. Setting
themselves to "done" and cascading the effect downstream.

This causes the pipeline to be done, but leaves a "filter" left-over that is not
complete. The consecuence is a "pile up" in `response_bus._slots` as filter
listeners remain connected. every new request creates a new slots entry. Which
is clearly not acceptable.

Proposed solution is to create a "notifier" event, and start the pipeline with
`takeuntil(notifier)` once we receive our results. we call `notifier.emit()` to
stop the pipeline from the starting point.
@q-phi
Copy link

q-phi commented Feb 27, 2026

This has been open for a while, are people mainly using ib_async with v1034 of TWS only?

From my research it appears that ib_async would be broken on anything above v1034, but I am not fully sure as I have not tested it myself.

@mattsta
Copy link
Contributor

mattsta commented Feb 27, 2026

Yeah, I'm currently running Gateway 10.40.1d with the regular primary branch here and it works as expected (i.e. no protobuf needed there yet).

The next big update for our project here will be more live testing/reviewing of all these protobuf re-architecture changes then branching off into a new major "protobuf-only" version going forward.

eta: ??? I'm still lagging behind a bit on the other updates for even 3.14 validation which we have workarounds/improvements for already, but haven't been merged back coherently yet into a packaged version update.

@q-phi
Copy link

q-phi commented Feb 27, 2026

Yeah, I'm currently running Gateway 10.40.1d with the regular primary branch here and it works as expected (i.e. no protobuf needed there yet).

The next big update for our project here will be more live testing/reviewing of all these protobuf re-architecture changes then branching off into a new major "protobuf-only" version going forward.

eta: ??? I'm still lagging behind a bit on the other updates for even 3.14 validation which we have workarounds/improvements for already, but haven't been merged back coherently yet into a packaged version update.

So the latest ib gateway works fine with the current version of ib_async (with no protobuf support), am i understanding you correctly? What about the latest version of TWS?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants