Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
d57ce36
risk analyzer scripts and changes required
abouchs-ravenpack Jun 27, 2025
7d402db
addressing comments and using excel manager to save
abouchs-ravenpack Jun 30, 2025
b025cea
Refactor settings to utils
jfaldanam Jul 1, 2025
4f56e38
Refactor excel related functions into the excel module
jfaldanam Jul 1, 2025
6567010
Remove unneded copyright notices
jfaldanam Jul 1, 2025
7bf0cf3
PX-550 Refactor themes submodule in a more generic tree structure
jfaldanam Jul 7, 2025
302231d
Conflict resolution with main
jfaldanam Sep 16, 2025
354aa69
proposed fixes to id-only logic
abouchs-ravenpack Sep 16, 2025
33a5b4b
Fix conflicts
jfaldanam Sep 16, 2025
bf020cd
Fix imports
jfaldanam Sep 16, 2025
e6c5a05
Fix docstring of screen_companies in risk analyzer
jfaldanam Sep 23, 2025
457b64f
Rename freq parameter to frequency for consistency with all workflows
jfaldanam Oct 2, 2025
af697d0
Change build system to properly install package locally
jfaldanam Oct 2, 2025
8dddce9
Fix CI filters and update changelog
jfaldanam Oct 2, 2025
bebe1f8
Remove old, outdated docs, documentation now lives at https://docs.bi…
jfaldanam Oct 2, 2025
3750f8e
Add formater, linter and type checker
jfaldanam Oct 2, 2025
134377e
Add CI pipeline for formatting, linting and running unit test
jfaldanam Oct 2, 2025
178e8bd
Ensure all non-optional dependencies are accuratly classified
jfaldanam Oct 2, 2025
13aa027
Disable tests on CI while the project is still not mature
jfaldanam Oct 2, 2025
05275eb
Add example and tutorials to linting and formatting process
jfaldanam Oct 2, 2025
82f5ee0
adding entity ticker and id to output df
abouchs-ravenpack Oct 3, 2025
1ccbd55
Migrate typing to modern types
jfaldanam Oct 3, 2025
b4e9987
Merge branch 'preparation_for_v1' into issue-20-miners-update
jfaldanam Oct 3, 2025
cd40f6d
Make fiscal year support several values
jfaldanam Oct 3, 2025
3a7bd35
Reduce number of type check errors and improve parameter validation
jfaldanam Oct 8, 2025
591e1cb
Use overload to fix typing issue
jfaldanam Oct 8, 2025
926b749
Remove type checker errors
jfaldanam Oct 9, 2025
36a99b1
Ignore all output excel files
jfaldanam Oct 9, 2025
fe28efc
Finish fixing type errors and enable again the ty step on the CI pipe…
jfaldanam Oct 9, 2025
d9ddeb6
Remove openai only exception control on generic module
jfaldanam Oct 9, 2025
755e3f6
Fix type errors
jfaldanam Oct 14, 2025
494f51b
Make examples output go to gitignored folder
jfaldanam Oct 14, 2025
28861cd
Implement custom motivations prompts for different use cases
jfaldanam Oct 14, 2025
1ca24da
Removed ipython dependency and unused code
jfaldanam Oct 14, 2025
02b9cf7
Change year start string representation after deprecation in pandas
jfaldanam Oct 16, 2025
31bb6a6
Sync with main after v0.20.2
jfaldanam Oct 22, 2025
74ccf0c
Fix type errors after merge with master and linting and formatting ne…
jfaldanam Oct 22, 2025
7e33ffa
Bump version
jfaldanam Oct 23, 2025
2e9324f
Fix tracing on v1 beta
jfaldanam Oct 23, 2025
4d1bb23
Fix very long delays on a very small number of OpenAI request
jfaldanam Oct 24, 2025
b3009b1
adding llmconfig to support reasoning models
abouchs-ravenpack Nov 5, 2025
c8f397e
Update pre commit hook
jfaldanam Nov 6, 2025
e14b9cf
removed redundant code and fixed typing
abouchs-ravenpack Nov 6, 2025
ce64eac
Merge pull request #25 from Bigdata-com/llm-reasoning-models
abouchs-ravenpack Nov 7, 2025
a96a6e9
fixed docstring
abouchs-ravenpack Nov 7, 2025
a2860ac
Merge pull request #21 from Bigdata-com/issue-20-miners-update
abouchs-ravenpack Nov 7, 2025
f590bc8
added support for source names
abouchs-ravenpack Nov 10, 2025
d58570f
Merge conflicts with v1 branch
jfaldanam Nov 10, 2025
694012e
returning topics and source details cols
abouchs-ravenpack Nov 10, 2025
fc28a06
improved typing and source handling
abouchs-ravenpack Nov 10, 2025
fb4c8ef
Big refactoring of query building logic
jfaldanam Nov 10, 2025
0d9dc41
Fix formatting and linting
jfaldanam Nov 11, 2025
3b5b825
Sync with v1 branch and formatting
jfaldanam Nov 11, 2025
10076b8
increasing llm tiemouts
abouchs-ravenpack Nov 11, 2025
b4d90d8
Merge pull request #19 from Bigdata-com/tech-debt-search
jfaldanam Nov 11, 2025
1f9c051
Error control and fix type issues
jfaldanam Nov 11, 2025
937cee4
Add connection configuration to LLM providers
jfaldanam Nov 11, 2025
23a9378
Expose llm connection config in llm config
jfaldanam Nov 11, 2025
e29088c
Sample data before doing motivation to avoid overloading the LLM context
jfaldanam Nov 11, 2025
8c0a3c2
Add timeout to llm config
jfaldanam Nov 12, 2025
ed69aad
added single response validation for risk labeler
abouchs-ravenpack Nov 12, 2025
28f8719
updated screener and narrative labeler
abouchs-ravenpack Nov 12, 2025
60a417e
handling and logging azure jailbreak errors
abouchs-ravenpack Nov 12, 2025
2f2386b
fixed sampling data before motivation
abouchs-ravenpack Nov 12, 2025
4b63505
other sampling fixes
abouchs-ravenpack Nov 12, 2025
cc778d7
more fixes
abouchs-ravenpack Nov 12, 2025
3164024
Formatting and linting
jfaldanam Nov 13, 2025
9871bc4
Pin version of type checker
jfaldanam Nov 13, 2025
68f47b7
Remove print in favor of loggers
jfaldanam Nov 13, 2025
b8f366a
Remove unneded logs
jfaldanam Nov 13, 2025
2f5b56e
Solve typing errors
jfaldanam Nov 13, 2025
f1864d3
Fix type issue
jfaldanam Nov 13, 2025
6d7fe23
Fix typing error
jfaldanam Nov 13, 2025
46750ea
Fixeed json serialization
jfaldanam Nov 14, 2025
7534410
Sync with master and resolve conflicts
jfaldanam Nov 17, 2025
f68a7b5
migrating deep mindmaps
abouchs-ravenpack Nov 17, 2025
5a0a50d
extracting sentence from query
abouchs-ravenpack Nov 17, 2025
ce09f9a
added presentation style visuals
abouchs-ravenpack Nov 18, 2025
2580cc0
fix query for grounding
abouchs-ravenpack Nov 18, 2025
e734073
Fix conflicts with v1 branch
jfaldanam Nov 18, 2025
c7e0461
Formatting and linting
jfaldanam Nov 18, 2025
7735425
robust grounding search
abouchs-ravenpack Nov 18, 2025
96c2217
improved typing and robust argument passing
abouchs-ravenpack Nov 18, 2025
c08938a
Improve typing
jfaldanam Nov 19, 2025
ed42c46
Remove all type errors
jfaldanam Nov 19, 2025
559ac3a
Format and linting
jfaldanam Nov 19, 2025
c4a8e97
grounding for workflows and motivation issues
abouchs-ravenpack Nov 19, 2025
d35ff85
removed max tokens for reasoning models
abouchs-ravenpack Nov 19, 2025
eda63c9
Fix type errors and formatting
jfaldanam Nov 19, 2025
9743bf3
Simplify default values
jfaldanam Nov 19, 2025
b9505f7
Merge pull request #29 from Bigdata-com/deep-mind-maps
jfaldanam Nov 19, 2025
4517458
Fix issue with rare, but existing deleted entities which cant be retr…
jfaldanam Nov 20, 2025
f984896
Merge branch 'preparation_for_v1' of github.com:Bigdata-com/bigdata-r…
jfaldanam Nov 20, 2025
510b278
Add common logic to many cookbooks back
jfaldanam Nov 21, 2025
3a6fe41
Bump version
jfaldanam Nov 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ on:
push:
tags:
- 'v*'
- '!v*-beta*'

jobs:
release:
Expand Down
48 changes: 48 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
name: Lint, format and run unit tests
on:
push:
branches:
- master
pull_request:

jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.11
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Install dependencies
run: uv sync --locked --dev
- name: Type check the code
run: make type-check
- name: Lint the code
run: make lint-check
- name: Format the code
run: make format
unit-tests:
runs-on: ubuntu-latest
needs: lint
strategy:
matrix:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install uv
uses: astral-sh/setup-uv@v6
with:
python-version: ${{ matrix.python-version }}
enable-cache: true
- name: Install dependencies
run: uv sync --locked --dev
# Disable type checking for now, v1 is not mature enough to pass all tests
# - name: Test with pytest
# run: make tests
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,10 @@ __pycache__/
dist/
build/
.ipynb_checkpoints

outputs/*
*.xlsx
*.xls

.coverage
coverage-reports/
28 changes: 19 additions & 9 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,12 +1,22 @@
# See https://pre-commit.com for more information
repos:
- repo: https://github.com/psf/black
rev: 24.1.1
- repo: local
hooks:
- id: black
- repo: https://github.com/pycqa/isort
rev: 5.13.2
hooks:
- id: isort
name: isort (python)
args: [--profile, black]
- id: make-lint
name: Execute Python linters
entry: make lint-check
language: system
pass_filenames: false
stages: [pre-commit]
- id: make-format
name: Format Python code
entry: make format-check
language: system
pass_filenames: false
stages: [pre-commit]
- id: make-type-check
name: Type check Python code
entry: make type-check
language: system
pass_filenames: false
stages: [pre-commit]
36 changes: 0 additions & 36 deletions .readthedocs.yaml

This file was deleted.

26 changes: 26 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,32 @@ file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.0.0] - 2025-11-21
Preparation for a first stable release.

## Added
- Added support for providing several fiscal years to any workflow or search function that
accepts a `fiscal_year` parameter. The parameter can now be a single integer or a list
of integers. When a list is provided, the workflow or function will search for the union
of all values provided.

### Changed
- Refactor the `themes` submodule to `tree`, allowing for a more generic tree structure
that can be re-used accross different workflows.
- Moved `workflows.utils.save_to_excel` to `excel.save_to_excel`, grouping all optional
features related to Excel in one place.
- Renamed `settings` to `utils` to better reflect its purpose as a utility module and free
the name for future use.
- Renamed all `freq` parameters to `frequency` for better clarity and consistency accross workflows.
- Implement custom motivation prompts for different use cases, including thematic screening and risk analysis.

### Fixed
- Fix duplicate dependencies in main vs optional dependencies. `openai` is now only optional while `graphviz`, `openpyxl` and `Pillow` is now only in main dependencies.

### Removed
- Removed support for Python 3.9 as it has reached its end of life. The minimum supported version is now Python 3.10.
- Removed `ipython` from main dependencies and removed unused function `bigdata_research_tools.workflows.utils.display_output_chunks_dataframe`.

## [0.21.1] - 2025-11-13

### Added
Expand Down
34 changes: 16 additions & 18 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,24 +1,22 @@
# Minimal makefile for Sphinx documentation
#
.PHONY: tests lint format

# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = docs
BUILDDIR = build
install-pre-commit:
@uvx pre-commit install

# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
tests:
@uv run -m pytest --cov --cov-report term --cov-report xml:./coverage-reports/coverage.xml -s tests/*

.PHONY: help Makefile
lint:
@uvx ruff check --extend-select I --fix src/bigdata_research_tools/ examples/ tutorial/ tests/

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
lint-check:
@uvx ruff check --extend-select I src/bigdata_research_tools/ examples/ tutorial/ tests/

format:
@uvx ruff format src/bigdata_research_tools/ examples/ tutorial/ tests/

create-docs:
@uv run $(MAKE) clean html
format-check:
@uvx ruff format --check bigdata_thematic_screener/ tests/

type-check:
@uvx ty@0.0.1a26 check --python-version 3.13 src/bigdata_research_tools/ examples/ tests/ # tutorial/ # Fix version to 3.13 due to this issue https://github.com/astral-sh/ty/issues/1355 # Ignore tutorials, the issues come from this open issue https://github.com/astral-sh/ty/issues/1297
13 changes: 11 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -354,7 +354,7 @@ Parameters to run the analysis end-to-end.
|-----------|------|---------|-------------|
| `document_limit` | `int` | `10` | Documents per query (see [Document Limit Guide](#document-limit))|
| `batch_size` | `int` | `10` | Batch size for processing (see [Batch Size Parameter Guide](#batch-size))|
| `freq` | `str` | `"3M"` | Date range frequency (see [Frequency Parameter Guide](#frequency))|
| `frequency` | `str` | `"3M"` | Date range frequency (see [Frequency Parameter Guide](#frequency))|
| `export_path` | `str` | `None` | Excel export path |

#### Return Values
Expand Down Expand Up @@ -477,7 +477,7 @@ results = run_search(
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `queries` | `List[QueryComponent]` | | List of search queries |
| `date_ranges` | `DATE_RANGE_TYPE` | `None` | Date range specifications |
| `date_ranges` | `INPUT_DATE_RANGE` | | Date range specifications |
| `limit` | `int` | `10` | Results per query |
| `only_results` | `bool` | `True` | Return format control |
| `scope` | `DocumentType` | `ALL` | Document type filter |
Expand Down Expand Up @@ -1198,6 +1198,15 @@ INFO:__main__: Bloomberg: 8 documents

---


## Tooling
This project uses [ruff](https://docs.astral.sh/ruff/) for linting and formatting and [ty](https://docs.astral.sh/ty/) for a type checker. To ensure your code adheres to the project's style guidelines, run the following commands before committing your changes:
```bash
make type-check
make lint
make format
```

## License

This software is licensed for use solely under the terms agreed upon in the
Expand Down
27 changes: 0 additions & 27 deletions docs/_static/bigdata_dark.svg

This file was deleted.

26 changes: 0 additions & 26 deletions docs/_static/bigdata_light.svg

This file was deleted.

9 changes: 0 additions & 9 deletions docs/bigdata_sdk/index.rst

This file was deleted.

2 changes: 0 additions & 2 deletions docs/changelog_link.md

This file was deleted.

Loading