-
Notifications
You must be signed in to change notification settings - Fork 0
Feature/ml stain normalization #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
LAdam-ix
wants to merge
80
commits into
main
Choose a base branch
from
feature/ml-stain-normalization
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+8,636
−1
Draft
Changes from all commits
Commits
Show all changes
80 commits
Select commit
Hold shift + click to select a range
0f02e50
feat: core structure
matejpekar 3c62cb6
feat: script for syncing project with template
matejpekar b0f97c1
fix: remove default "exp" type from commit
matejpekar 2a22665
fix: remove silence
matejpekar 13f7425
fix: ignore mlruns
matejpekar b8db313
fix: remove empty line between import types
matejpekar 6f290ec
feat: separate test dependencies
matejpekar 2b5660f
feat: CI/CD
matejpekar 69357fc
fix: pytest allow empty tests
matejpekar da66245
feat: syncing pipeline + google code styles
matejpekar 97e7697
fix: pipeline rules
matejpekar 225ee77
fix: pylint max line length 88
matejpekar 56cdb9c
refactor: new bot user
matejpekar 0dd2b04
chore: format yaml
matejpekar 9ed4f98
feat: docstring code format
matejpekar c62740a
fix: update pre-commit versions
matejpekar ff95fb8
feat: test coverage
matejpekar 603ab7d
feat: allow installing gitlab dependecies in CI/CD
matejpekar cdc32c1
chore: ignore mlflow_store
matejpekar 584cdf2
chore: new libary template
matejpekar 168b795
feat: routines for separating and combining stain channels, and stain…
TomasBalak 4b5b907
feat: convert_color with sequence
matejpekar 3ff13a4
docs: reorganize
matejpekar a10335a
fix: py.typed marker file
matejpekar bd1d704
feat: light H&DAB protocol, color conversion fixes, modify and recons…
TomasBalak b8570a0
feat: ml init
LAdam-ix 844d765
fix: transformations inputs
fc427f8
chore: added data link
f039733
fix: resolve issues before job run
9c6aad7
chore: job config
a593c29
fix: job run parametrs
4c9e89d
fix: lower batch size
b4df4e3
fix: lower workers
4af7ac1
fix: ensure float32 tensors
07a4002
fix: missing ()
1bc166c
fix: decode_head settings
c258af2
chore: config change
8d8190d
fix: job settign optimalization
d9e3319
fix: image normalization
986667d
feat: tiles export
834a149
feat: update loss function
6c02839
fix: p=1
68c919d
fix: more frequent validation
a2a5725
fix: val logging
497328c
fix: val logging
80b4773
fix: repair loss function
9d60be8
fix: revert val logging
20caad9
fix: optimalise train data load
4da48cd
fix: update val frequency
0df039e
feat: change to full unet
c000c35
chore: update job setup
8522a27
chore: update job setup
ad4bfd1
chore: code cleanup
8beb398
chore: formating fix
082e5e1
chore: formating fix
137fa37
chore: formating fix
7d6aee2
chore: remove kubejobs
0541142
chore: naming fix
b3cc698
chore: add comments and fix formating
LAdam-ix 1bbce80
feat: added demo
LAdam-ix a36d502
fix: formating
LAdam-ix cdd39a6
fix: arguments
b8a928c
feat: add brightness loss and analysis
8142cc7
feat: update dataset slide paths
b6341fe
fix: update dataset path
LAdam-ix de4e59b
fix: update dataset path
LAdam-ix 3584636
fix: update dataset path
LAdam-ix b986f3c
chore: change batch size
LAdam-ix 87dca95
fix: correct python version
LAdam-ix aaa8d3b
fix: add missing import
LAdam-ix d500e91
chore: change batch size
LAdam-ix 0436665
chore: change batch size
LAdam-ix 1896269
fix: adjust hyperparameters
LAdam-ix 9c31f5a
chore: add checkpoint path
LAdam-ix 03791ff
feat: add brightness los
LAdam-ix e6c19cb
fix: add missed line
LAdam-ix a6e54ee
fix: dimension in brightness loss
LAdam-ix 4bd95d4
chore: update checkpoint
9f5ab8a
feat: update analysis methods and WSI assembly
LAdam-ix 0ab457f
chore: merge main history
LAdam-ix File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| [flake8] | ||
| max-line-length=88 | ||
| extend-ignore=E203 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,183 @@ | ||
| # Byte-compiled / optimized / DLL files | ||
| __pycache__/ | ||
| *.py[cod] | ||
| *$py.class | ||
|
|
||
| # C extensions | ||
| *.so | ||
|
|
||
| # Distribution / packaging | ||
| .Python | ||
| build/ | ||
| develop-eggs/ | ||
| dist/ | ||
| downloads/ | ||
| eggs/ | ||
| .eggs/ | ||
| lib/ | ||
| lib64/ | ||
| parts/ | ||
| sdist/ | ||
| var/ | ||
| wheels/ | ||
| share/python-wheels/ | ||
| *.egg-info/ | ||
| .installed.cfg | ||
| *.egg | ||
| MANIFEST | ||
|
|
||
| # PyInstaller | ||
| # Usually these files are written by a python script from a template | ||
| # before PyInstaller builds the exe, so as to inject date/other infos into it. | ||
| *.manifest | ||
| *.spec | ||
|
|
||
| # Installer logs | ||
| pip-log.txt | ||
| pip-delete-this-directory.txt | ||
|
|
||
| # Unit test / coverage reports | ||
| htmlcov/ | ||
| .tox/ | ||
| .nox/ | ||
| .coverage | ||
| .coverage.* | ||
| .cache | ||
| nosetests.xml | ||
| coverage.xml | ||
| *.cover | ||
| *.py,cover | ||
| .hypothesis/ | ||
| .pytest_cache/ | ||
| cover/ | ||
|
|
||
| # Translations | ||
| *.mo | ||
| *.pot | ||
|
|
||
| # Django stuff: | ||
| *.log | ||
| local_settings.py | ||
| db.sqlite3 | ||
| db.sqlite3-journal | ||
|
|
||
| # Flask stuff: | ||
| instance/ | ||
| .webassets-cache | ||
|
|
||
| # Scrapy stuff: | ||
| .scrapy | ||
|
|
||
| # Sphinx documentation | ||
| docs/_build/ | ||
|
|
||
| # PyBuilder | ||
| .pybuilder/ | ||
| target/ | ||
|
|
||
| # Jupyter Notebook | ||
| .ipynb_checkpoints | ||
|
|
||
| # IPython | ||
| profile_default/ | ||
| ipython_config.py | ||
|
|
||
| # pyenv | ||
| # For a library or package, you might want to ignore these files since the code is | ||
| # intended to run in multiple environments; otherwise, check them in: | ||
| # .python-version | ||
|
|
||
| # pipenv | ||
| # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. | ||
| # However, in case of collaboration, if having platform-specific dependencies or dependencies | ||
| # having no cross-platform support, pipenv may install dependencies that don't work, or not | ||
| # install all needed dependencies. | ||
| #Pipfile.lock | ||
|
|
||
| # poetry | ||
| # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. | ||
| # This is especially recommended for binary packages to ensure reproducibility, and is more | ||
| # commonly ignored for libraries. | ||
| # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control | ||
| #poetry.lock | ||
|
|
||
| # pdm | ||
| # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. | ||
| #pdm.lock | ||
| # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it | ||
| # in version control. | ||
| # https://pdm-project.org/#use-with-ide | ||
| .pdm.toml | ||
| .pdm-python | ||
| .pdm-build/ | ||
|
|
||
| # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm | ||
| __pypackages__/ | ||
|
|
||
| # Celery stuff | ||
| celerybeat-schedule | ||
| celerybeat.pid | ||
|
|
||
| # SageMath parsed files | ||
| *.sage.py | ||
|
|
||
| # Environments | ||
| .env | ||
| .venv | ||
| env/ | ||
| venv/ | ||
| ENV/ | ||
| env.bak/ | ||
| venv.bak/ | ||
|
|
||
| # Spyder project settings | ||
| .spyderproject | ||
| .spyproject | ||
|
|
||
| # Rope project settings | ||
| .ropeproject | ||
|
|
||
| # mkdocs documentation | ||
| site | ||
|
|
||
| # mypy | ||
| .mypy_cache/ | ||
| .dmypy.json | ||
| dmypy.json | ||
|
|
||
| # ruff | ||
| .ruff_cache/ | ||
|
|
||
| # Pyre type checker | ||
| .pyre/ | ||
|
|
||
| # pytype static type analyzer | ||
| .pytype/ | ||
|
|
||
| # Cython debug symbols | ||
| cython_debug/ | ||
|
|
||
| # VSCode | ||
| .vscode/ | ||
|
|
||
| # MacOS | ||
| .DS_Store | ||
|
|
||
| # Mlflow | ||
| mlruns/ | ||
| mlflow_store/ | ||
| mlflow_artifacts/ | ||
|
|
||
| # Pytorch Lightning | ||
| lightning_logs/ | ||
|
|
||
| # PyCharm | ||
| # JetBrains specific template is maintained in a separate JetBrains.gitignore that can | ||
| # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore | ||
| # and can be added to the global gitignore or merged into this file. For a more nuclear | ||
| # option (not recommended) you can uncomment the following to ignore the entire idea folder. | ||
| .idea/ | ||
|
|
||
| # Project | ||
| /data | ||
| /demo_* |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| include: | ||
| project: rationai/digital-pathology/templates/ci-templates | ||
| file: Python-Lint.gitlab-ci.yml | ||
|
|
||
| stages: | ||
| - lint |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| [mypy] | ||
| strict = True | ||
| ignore_missing_imports = True | ||
| disallow_untyped_calls = False | ||
| disable_error_code = no-any-return |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # See https://pre-commit.com for more information | ||
| # See https://pre-commit.com/hooks.html for more hooks | ||
| repos: | ||
| - repo: https://github.com/pre-commit/pre-commit-hooks | ||
| rev: v5.0.0 | ||
| hooks: | ||
| - id: check-yaml | ||
| args: [--unsafe] | ||
|
|
||
| - repo: https://github.com/commitizen-tools/commitizen | ||
| rev: v3.30.1 | ||
| hooks: | ||
| - id: commitizen | ||
|
|
||
| - repo: https://github.com/astral-sh/ruff-pre-commit | ||
| rev: v0.7.3 | ||
| hooks: | ||
| # Run the linter. | ||
| - id: ruff | ||
| entry: pdm lint --force-exclude | ||
| # Run the formatter. | ||
| - id: ruff-format | ||
| entry: pdm format --force-exclude |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| fix = true | ||
| line-length = 88 | ||
| target-version = "py311" | ||
|
|
||
| [format] | ||
| # Enable reformatting of code snippets in docstrings. | ||
| docstring-code-format = true | ||
|
|
||
| [lint] | ||
| unfixable = [ | ||
| "ERA", # do not autoremove commented out code | ||
| ] | ||
| extend-select = [ | ||
| "B", # flake8-bugbear | ||
| "C4", # flake8-comprehensions | ||
| "ERA", # flake8-eradicate/eradicate | ||
| "I", # isort | ||
| "N", # pep8-naming | ||
| "PIE", # flake8-pie | ||
| "PGH", # pygrep | ||
| "RUF", # ruff checks | ||
| "SIM", # flake8-simplify | ||
| "TCH", # flake8-type-checking | ||
| "TID", # flake8-tidy-imports | ||
| "UP", # pyupgrade | ||
| "D", # pydocstyle | ||
| ] | ||
| extend-ignore = [ | ||
| "ERA001", # commented out code | ||
| "D100", # missing docstring in public module | ||
| "D101", # missing docstring in public class | ||
| "D102", # missing docstring in public method | ||
| "D103", # missing docstring in public function | ||
| "D104", # missing docstring in public package | ||
| "D105", # missing docstring in magic method | ||
| "D106", # missing docstring in public nested class | ||
| "D107", # missing docstring in __init__ | ||
| "N812", # lowercase imported as non lowercase | ||
| "TCH002", # move third-party into a type-checking block | ||
| "F722", # jaxtyping | ||
| ] | ||
|
|
||
| [lint.flake8-tidy-imports] | ||
| ban-relative-imports = "all" | ||
|
|
||
| [lint.isort] | ||
| lines-after-imports = 2 | ||
| known-first-party = ["tests"] | ||
|
|
||
| [lint.pydocstyle] | ||
| convention = "google" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,34 @@ | ||
| # stain-normalization | ||
| # Stain Normalization | ||
|
|
||
| Tento repozitár slúži ako doplnkový materiál k bakalárskej práci **"Normalizácia farbenia histopatologických snímkov pomocou neuronových sietí"**. | ||
|
|
||
| Celý kód nie je možné spustiť samostatne, pretože vyžaduje prístup k citlivým dátam a k platforme na správu strojového učenia MLflow. Avšak je možné spustiť demo, ku ktorému je pripravených pár vzoriek na demonštráciu. | ||
|
|
||
| ## Demo | ||
|
|
||
| Demo skript umožňuje načítať jeden obrázok alebo celý priečinok, normalizovať ich pomocou predtrénovaného modelu a uložiť výsledné obrázky do určeného priečinka. | ||
|
|
||
| ## Priložené dáta | ||
|
|
||
| - **Originálne obrázky** (v priečinku `./demo_data/original`) – referenčné vzorky. | ||
| - **Upravené obrázky** (v priečinku `./demo_data/modified`) – referenčné obrázky s modifikovaným sfarbením, ktoré sa následne modelom znormalizujú späť k originálnemu vzhľadu. | ||
| - **Obrázky z iného datasetu** (v priečinku `./demo_data/to_predict`) – ukazujú, ako model dokáže prispôsobiť farby vstupov tak, aby zodpovedali farebnému štýlu trénovacích dát. | ||
| - **Model checkpoint** | ||
|
|
||
| ## Spustenie dema | ||
|
|
||
| Projekt je spravovaný pomocou nástroja **pdm**, ktorý umožňuje jednoduchú správu závislostí a prostredia. Stačí nainštalovať závislosti pomocou: | ||
|
|
||
| ```bash | ||
| pdm install | ||
| ``` | ||
| a potom spustiť demo skript: | ||
|
|
||
| ```bash | ||
| pdm run python demo.py --input ./demo_data/modified | ||
| ``` | ||
|
|
||
| ## Dostupné arguemnty: | ||
| - **input**: cesta k obrázku alebo priečinku s obrázkami na normalizáciu (default ./demo_data/modified) | ||
| - **output**: priečinok, kam sa uložia normalizované obrázky (default ./demo_data) | ||
| - **use_cpu**: defaultne nadstavené na použitie GPU ak je dostupná, avšak ak by nastali problémy odporúčam použivať iba CPU | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are a few typos in this section that affect readability. Specifically:
arguemntyshould beargumentynadstavenéshould benastavenépouživaťshould bepoužívať