Skip to content

Merge develop into main for next release #140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
philipp-fischer merged 322 commits into from
Jun 5, 2025
Merged

Merge develop into main for next release #140

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
322 commits
Select commit Hold shift + click to select a range
130b471
Add watchdog feature
philipp-fischer Apr 9, 2025
a6c3be6
Add license header
philipp-fischer Apr 9, 2025
37b1664
Initialize fastseek in the constructor
Queuecumber Apr 9, 2025
735cdad
Merge pull request #112 from Queuecumber/mehrlich/fastseek-fallback-l…
philipp-fischer Apr 10, 2025
e0fc5d3
Fix unit tests
philipp-fischer Apr 10, 2025
7906e38
Fix unit test with dataset config
philipp-fischer Apr 10, 2025
9962a8f
Address review
philipp-fischer Apr 10, 2025
e4aa5c1
Merge remote-tracking branch 'origin/develop' into feature/watchdog
philipp-fischer Apr 10, 2025
ab43da0
Add args
philipp-fischer Apr 10, 2025
2b77bee
Address remaining review comments
philipp-fischer Apr 10, 2025
3d5f444
Merge pull request #113 from NVIDIA/feature/watchdog
voegtlel Apr 10, 2025
0c29e3e
- Update cooker to support aux file stores and out-caching.
voegtlel Apr 11, 2025
e8f16fc
Fix decoder in val mode
voegtlel Apr 11, 2025
a995fa1
Fix packing at dataset exhaustion #114
voegtlel Apr 11, 2025
4f28339
Implement tests for cache pool. Not all are working yet already.
voegtlel Apr 11, 2025
8385764
Merge pull request #115 from NVIDIA/fix/packing_exhaustion
philipp-fischer Apr 14, 2025
54c7273
Merge branch 'develop' into feature/samples_on_demand
philipp-fischer Apr 14, 2025
d9d71c2
Move cache to get_*_loader instead of get_*_dataset
voegtlel Apr 14, 2025
1d35889
Fix tests for file cache pool
voegtlel Apr 14, 2025
57731bf
Introduce SystemFileStore
philipp-fischer Apr 14, 2025
1838a7e
Merge remote-tracking branch 'origin/feature/system_file_store' into …
voegtlel Apr 14, 2025
2425a8a
Introduce pre_/post_encode for use with packing.
voegtlel Apr 14, 2025
fc3a222
Fix tests, adapt defaults
voegtlel Apr 15, 2025
835eb21
Improve typing for epath and add read_bytes
voegtlel Apr 7, 2025
8448e40
Add test for same audio clip length
philipp-fischer Apr 16, 2025
8199dba
Option for get_single_frames_uniform to return timestamps of frames
philipp-fischer Apr 16, 2025
e9528a4
AVDecoder: decode avi, new methods, adjust audio clip length if close
philipp-fischer Apr 16, 2025
e5e4ff5
Fix a rounding bug in avdecoder
philipp-fischer Apr 16, 2025
0358356
Address review
philipp-fischer Apr 16, 2025
307616b
Fix test
philipp-fischer Apr 16, 2025
f4f8b53
Merge pull request #117 from NVIDIA/feature/av_fixes
philipp-fischer Apr 16, 2025
d8ee7bc
Update global exports
voegtlel Apr 17, 2025
ad9ddf5
Add system file store to metadataset and allow direct string config
voegtlel Apr 17, 2025
7e754f4
Separate audio and video duration
philipp-fischer Apr 17, 2025
5b20ec1
Remove debug print
philipp-fischer Apr 17, 2025
f31901a
Fix Last frame offset in get_clips_uniform
philipp-fischer Apr 17, 2025
ab5b5be
Optimization
philipp-fischer Apr 17, 2025
97f5707
Rename test
philipp-fischer Apr 17, 2025
c40d162
Cosmetic changes and exports
voegtlel Apr 17, 2025
50ee5a8
Fix threads before fork issue
philipp-fischer Apr 22, 2025
2fb6d28
Forgot some changes
philipp-fischer Apr 22, 2025
a848627
More verbose CI unit tests with per-test headers
philipp-fischer Apr 22, 2025
27b5526
Address review
philipp-fischer Apr 22, 2025
ab40226
Merge pull request #118 from NVIDIA/feature/improve_av_get_duration
philipp-fischer Apr 22, 2025
42fc350
Fix JoinDatasetLoader decoder arg
voegtlel Apr 22, 2025
7c90b8d
Fix test for removed ignore_decoder_errors
voegtlel Apr 22, 2025
66e0c27
Fix message for missing cooker
voegtlel Apr 22, 2025
84b480a
Implement __del__ for FileCacheLazy to ensure local disk space is rel…
voegtlel Apr 22, 2025
1fca43a
Fix MKV
philipp-fischer Apr 23, 2025
dd428da
Remove unused var
philipp-fischer Apr 23, 2025
cb507ad
Address some review comments
philipp-fischer Apr 23, 2025
2502f87
Add docs for MockLazy
voegtlel Apr 23, 2025
1c73a66
Add docs for MockLazy
voegtlel Apr 23, 2025
ce505f0
Change aux paths. Improve some docstrings
philipp-fischer Apr 24, 2025
72c697d
Fix unit test
philipp-fischer Apr 24, 2025
3431714
Fix FileCacheLazy.__del__ missing lock
voegtlel Apr 24, 2025
f51a143
Update data flow diagram
philipp-fischer Apr 24, 2025
1a4e5a6
Fix restore key for packing encode sample
voegtlel Apr 24, 2025
1017a7b
fix ruff
voegtlel Apr 24, 2025
6a2ce3e
Improved docs for pre/post encode
philipp-fischer Apr 24, 2025
11b626f
change protocol for filesystem to filesystem
voegtlel Apr 24, 2025
957e690
Make Lazy hashable, fix error printing
voegtlel Apr 25, 2025
75be0e0
Fix test
voegtlel Apr 25, 2025
4f7dd5c
Implement a method to get the source files for a sample
voegtlel Apr 25, 2025
02719b8
just fixed
voegtlel Apr 25, 2025
4df6c74
Allow legacy error handlers
voegtlel Apr 25, 2025
b41867b
Implement `__sources__` instead of separate method for sample source …
voegtlel Apr 25, 2025
ee79be0
Add test for sources and fix reader source
voegtlel Apr 25, 2025
06df87f
Throw exception on container parser not finding keyframes
Queuecumber Apr 28, 2025
5d3df6d
Fallback on any exception from fastseek parsing
Queuecumber Apr 28, 2025
10a1ffc
lint errors
Queuecumber Apr 28, 2025
329ac74
Fix watchdog local prints
voegtlel Apr 29, 2025
a2bb237
Move watchdog dataset to loader
philipp-fischer Apr 29, 2025
da99d5f
Improve typing for epath and add read_bytes
voegtlel Apr 7, 2025
c8dd2b7
Add test for same audio clip length
philipp-fischer Apr 16, 2025
e1c3d4f
Option for get_single_frames_uniform to return timestamps of frames
philipp-fischer Apr 16, 2025
1b0e7a8
AVDecoder: decode avi, new methods, adjust audio clip length if close
philipp-fischer Apr 16, 2025
0608d8d
Fix a rounding bug in avdecoder
philipp-fischer Apr 16, 2025
b30ad3b
Address review
philipp-fischer Apr 16, 2025
a19ab26
Fix test
philipp-fischer Apr 16, 2025
bda4e6b
More verbose CI unit tests with per-test headers
philipp-fischer Apr 22, 2025
a27ab19
Separate audio and video duration
philipp-fischer Apr 17, 2025
849efa1
Remove debug print
philipp-fischer Apr 17, 2025
1e968d4
Fix Last frame offset in get_clips_uniform
philipp-fischer Apr 17, 2025
e566f48
Optimization
philipp-fischer Apr 17, 2025
9e679ea
Rename test
philipp-fischer Apr 17, 2025
18e39d2
Fix threads before fork issue
philipp-fischer Apr 22, 2025
5be4ff5
Forgot some changes
philipp-fischer Apr 22, 2025
e0a4ef2
Address review
philipp-fischer Apr 22, 2025
60e596d
Make sure codec context is set for a stream before changing the threa…
Queuecumber Apr 29, 2025
0dd2e66
formatting
Queuecumber Apr 29, 2025
a6fcb20
Fix forking the file cache pool. Fix watchdog args. Add progbar to pa…
voegtlel Apr 29, 2025
1d839ef
Make sure to reset the stream pointer before trying probe mode
Queuecumber Apr 29, 2025
01661aa
formatting
Queuecumber Apr 29, 2025
57ba00a
Remove debug prints
voegtlel Apr 29, 2025
996c172
Add more explicit error conditions
Queuecumber Apr 29, 2025
1bdc929
Catch a bunch of exceptions explicitly
Queuecumber Apr 29, 2025
a4a53c0
Move handling of specific exceptions to parser logic
Queuecumber Apr 29, 2025
3455ce2
remove unused import
Queuecumber Apr 29, 2025
28e76c8
Merge pull request #123 from NVIDIA/feature/fastseek_more_fallbacks
philipp-fischer Apr 30, 2025
0b25a66
Merge remote-tracking branch 'origin/develop' into feature/fix_mkv_de…
philipp-fischer Apr 30, 2025
32a0d8c
Add warnings if seeker unit does not match
philipp-fischer Apr 30, 2025
afae33c
Fix fork hook
voegtlel Apr 30, 2025
ea89a0e
Fix after fork and fix internal pending prefetches
voegtlel Apr 30, 2025
e9b17a4
Don't print CUDA tensors in watchdog trace back
philipp-fischer Apr 30, 2025
363192d
Typo
philipp-fischer Apr 30, 2025
feb3bd5
Restore option to use energon without AV
philipp-fischer Apr 30, 2025
3822e90
Introduce faster yaml loading
philipp-fischer Apr 30, 2025
3bc4c97
Remove assertion
philipp-fischer Apr 30, 2025
60a9813
Add missing import
philipp-fischer Apr 30, 2025
753cf00
Some cleanup and docstring
philipp-fischer Apr 30, 2025
46829e6
Fix resource close issue
voegtlel Apr 30, 2025
314d66f
Update uv.lock
philipp-fischer May 2, 2025
fd39b10
Make json new default for info config format
philipp-fischer May 5, 2025
a8d237a
Missing import
philipp-fischer May 5, 2025
09c7188
Fix small issues, cleanup comments, implement fallback to split.json
voegtlel May 5, 2025
ab6861b
Remove strange comment
philipp-fischer May 6, 2025
79f6b7f
If .info.yaml was there, keep it but update it
philipp-fischer May 6, 2025
a23527f
Introduce bracecollapse
philipp-fischer May 7, 2025
473c710
Add missing license
philipp-fischer May 7, 2025
29397ae
Adjust code style
philipp-fischer May 7, 2025
17a5cc7
Adapt info command
philipp-fischer May 7, 2025
29d2955
Fix energon info for brace expand
philipp-fischer May 7, 2025
e2aea54
Remove unused code and fix inline comment
philipp-fischer May 7, 2025
6784bcc
Fix lint tool
philipp-fischer May 7, 2025
7479dcc
Merge pull request #124 from NVIDIA/feature/rapidyaml
voegtlel May 7, 2025
c2a7863
Improve how user-overridden methods are check in TaskEncoder and fix …
philipp-fischer May 7, 2025
6a9b8bd
Remove duplicate method
philipp-fischer May 7, 2025
25eef23
Fix bug in attribute check
philipp-fischer May 8, 2025
7251cee
Simplify override check
philipp-fischer May 8, 2025
2d16d8e
Polish
philipp-fischer May 8, 2025
9315d1e
Merge pull request #125 from NVIDIA/feature/improve_user_override_check
philipp-fischer May 8, 2025
c1e341b
Merge pull request #120 from NVIDIA/feature/fix_mkv_decoding
philipp-fischer May 8, 2025
e15ac04
Merge remote-tracking branch 'origin/develop' into feature/samples_on…
philipp-fischer May 8, 2025
c8d3520
Use new `_is_overridden`
philipp-fischer May 8, 2025
68244e5
Some fixes after merge
philipp-fischer May 8, 2025
de69fa2
Fix .info.yaml bug
philipp-fischer May 8, 2025
c03691b
Improve _is_overridden
philipp-fischer May 8, 2025
389aaf6
Fix save/restore with non divisible iterations vs num workers
voegtlel May 8, 2025
39533b2
Fix next_worker_id, hacky improve test speed
voegtlel May 8, 2025
d92fde7
fork hook efficiency
voegtlel May 8, 2025
401e6dc
Merge branch 'feature/fix_save_restore' into feature/samples_on_demand
voegtlel May 8, 2025
5c5a26d
Fix typing
voegtlel May 8, 2025
62a54ba
Merge commit '5c5a26db54bb119e342c8b3e24f48f0b572d7e19' into feature/…
voegtlel May 8, 2025
8675c7d
Fix bug by have `_workers_restore_from` always set. None can be per-w…
philipp-fischer May 9, 2025
7672954
Add test for initial state save/restore and fix code
voegtlel May 9, 2025
1582c17
Merge branch 'feature/fix_save_restore' into feature/samples_on_demand
voegtlel May 9, 2025
55a6db2
Merge pull request #126 from NVIDIA/feature/fix_save_restore
philipp-fischer May 9, 2025
6f52019
Fix handling of blend_epochized with repeat=True
voegtlel May 9, 2025
41a52da
Add repetitions for repeat=True and epochized_blend
voegtlel May 9, 2025
2920f94
Add tests for previous failure and move shufflebuffer to outside for …
voegtlel May 9, 2025
6d9fa07
Improve docstring as discussed
philipp-fischer May 9, 2025
72fbba8
Implement filetype guessing decoder
voegtlel May 12, 2025
de1b461
Fix sample decoder
voegtlel May 13, 2025
b2c127d
Fix test
voegtlel May 13, 2025
387da70
Remove unused `id` field from sqlite
philipp-fischer May 13, 2025
14203c8
Merge pull request #111 from NVIDIA/feature/samples_on_demand
philipp-fischer May 13, 2025
6aa0a75
Merge remote-tracking branch 'origin/develop' into feature/sample_source
philipp-fischer May 13, 2025
3c93492
Remove @dataclass_slots fallback
philipp-fischer May 13, 2025
6182e9a
Sample with default-value for `__sources__` (and also subflavors)
philipp-fischer May 13, 2025
09e9247
Adapt uv.lock to Py 3.10
philipp-fischer May 13, 2025
aa484fa
Add missing license header
philipp-fischer May 13, 2025
f9ac597
Pin test python version
philipp-fischer May 13, 2025
c4f9a1b
Add @edataclass
philipp-fischer May 14, 2025
4409e79
Remove unused imports
philipp-fischer May 14, 2025
ec3070d
Fix many unit tests
philipp-fischer May 14, 2025
55f7bf1
A first PoC of the `energon mount` command
philipp-fischer May 14, 2025
55d2f1f
Ruff
philipp-fischer May 14, 2025
4a8fd7c
Fix ls bug (slash prefix)
philipp-fischer May 14, 2025
2346971
Fix test for source
voegtlel May 15, 2025
1e5e3d1
Small rename and restructure
philipp-fischer May 15, 2025
3a33fb2
Fix source info in tests
voegtlel May 15, 2025
e70bbec
Working energon prepare with new sample_parts table
philipp-fischer May 15, 2025
a886c5f
Store content byte offsets
philipp-fischer May 15, 2025
38ce13a
First version of mount seems to be functional for basic datasets.
philipp-fischer May 15, 2025
f6e3688
Fix mount bug (sample shifted)
philipp-fischer May 15, 2025
b1e4760
Replace ratarmount with mfusepy
philipp-fischer May 16, 2025
6004b45
Exit gracefully
philipp-fischer May 16, 2025
a8edc32
Fix sqlite close and upgrade to Py3.10
philipp-fischer May 16, 2025
2456cd2
Improve the mount feature and add sample-folder mode
philipp-fischer May 22, 2025
0d2a60a
Make `energon mount` work with remote FS
philipp-fischer May 23, 2025
b6f2688
Remove print
philipp-fischer May 23, 2025
e229852
Add docs for `energon mount`
philipp-fischer May 26, 2025
3fd40b4
Revert debug change in test_dataset.py
philipp-fischer May 26, 2025
1dd6142
Fix deprecated decoder args
philipp-fischer May 26, 2025
43cd18f
Docs for new sample decoding
philipp-fischer May 26, 2025
5d1c1e5
Adapt task encoder docs
philipp-fischer May 26, 2025
d9d575f
Intermediate
philipp-fischer May 26, 2025
38caa7e
Add note on remote datasets
philipp-fischer May 26, 2025
b66bed6
Merge pull request #121 from NVIDIA/feature/sample_source
philipp-fischer May 27, 2025
4f23486
Merge remote-tracking branch 'origin/develop' into feature/samples_on…
philipp-fischer May 27, 2025
e56789a
Fix edataclass typing
voegtlel May 27, 2025
ba5428e
Implement batch and mapping failure if 100 samples in sequence fail
voegtlel May 27, 2025
24cf820
Extend `Batch` base class with default attributes from `Sample`
voegtlel May 27, 2025
b02999c
Fix for py3.10
voegtlel May 27, 2025
cb85ff1
Update docs
voegtlel May 27, 2025
9be4dd4
Fix av decoder in case of no valid frames
philipp-fischer May 27, 2025
cafd70a
Simplify frame counting
philipp-fischer May 27, 2025
814baa2
Hopefully fix now
philipp-fischer May 27, 2025
0f43280
Merge pull request #132 from NVIDIA/feature/fix_av_decoder
philipp-fischer May 27, 2025
6f08a49
Make failure tolerance configurable
voegtlel May 27, 2025
70de319
Add energon mount slow mode
philipp-fischer May 27, 2025
83e1b6c
Remove debug prints
philipp-fischer May 27, 2025
ae4bfa8
More debug prints
philipp-fischer May 27, 2025
e758041
Merge remote-tracking branch 'origin/develop' into feature/mount
philipp-fischer May 27, 2025
b7d405b
Fix ruff
philipp-fischer May 27, 2025
ccd491f
Apply fix as dicussed
philipp-fischer May 27, 2025
f999a2a
Move __subflavor__ into __subflavors__ and replace by deprecated prop…
voegtlel May 27, 2025
c413080
Completely remove `Sample.__subflavor__` and `Batch.__subflavor__`. R…
voegtlel May 27, 2025
2f0fbdf
Merge remote-tracking branch 'origin/develop' into feature/fix_batch_…
voegtlel May 27, 2025
ada36ec
Reformat
voegtlel May 27, 2025
08471f7
Merge pull request #131 from NVIDIA/feature/fix_batch_basecls
voegtlel May 27, 2025
622895d
Merge remote-tracking branch 'origin/develop' into feature/mount
voegtlel May 27, 2025
5044ae5
Update commends
voegtlel May 28, 2025
d0bcaf1
Replace torch.multinomial by custom implementation. Verified distribu…
voegtlel May 28, 2025
b2bce40
Fix epath test for s3
voegtlel May 28, 2025
0746e22
uv lock update
voegtlel May 28, 2025
accfc47
Fix typing for mount
voegtlel May 28, 2025
a4f45b0
Optimize speed by removing TypedDict as requested
voegtlel May 28, 2025
b746421
Merge pull request #129 from NVIDIA/feature/mount
philipp-fischer May 30, 2025
f6ef96f
Implement s3 emulator for testing
voegtlel May 30, 2025
cfde9c1
Fix EPath handling for paths with protocol (e.g. absolute msc:// paths)
voegtlel May 30, 2025
14973c1
Add fix license header to just fix and fix now
voegtlel May 30, 2025
31e0e1e
Try fix the unit test by waiting for S3 Emulator to be ready
philipp-fischer May 30, 2025
df6850b
Merge fixes from debug branch
philipp-fischer May 30, 2025
0563c33
Fix `mount` given new __sources__ adaptations
philipp-fischer May 30, 2025
80bed0b
Adapt (doc) fomatting to our guideline
voegtlel Jun 2, 2025
8f39e4b
Remove wait_until_ready, as it's not needed
voegtlel Jun 2, 2025
19c3bdf
Fail-safe fuse import
philipp-fischer Jun 2, 2025
99cfdbb
Fix S3 emulator
voegtlel Jun 2, 2025
5a46a5c
Merge pull request #135 from NVIDIA/feature/s3_tests_pathfix
voegtlel Jun 3, 2025
aa60854
Merge pull request #137 from NVIDIA/feature/fix_mount
voegtlel Jun 3, 2025
2a49c40
Merge branch 'develop' into feature/samples_on_demand_docs
philipp-fischer Jun 3, 2025
13e5e19
Fix docstrings for sphinx
philipp-fischer Jun 3, 2025
4c1eacc
Add pip extras
philipp-fischer Jun 3, 2025
8bd181e
Add glossary
philipp-fischer Jun 4, 2025
c2b4fd1
Improve dataprep
philipp-fischer Jun 4, 2025
7c9289f
Deprecate sample loader
philipp-fischer Jun 4, 2025
f6d0009
Auxiliary data and cache pools
philipp-fischer Jun 4, 2025
5b98766
Fix ref
philipp-fischer Jun 4, 2025
6a4d37c
Markdown fix
philipp-fischer Jun 4, 2025
6747bd7
Remove cooker condition
philipp-fischer Jun 5, 2025
3144cf9
Address review
philipp-fischer Jun 5, 2025
be57d6c
Merge pull request #139 from NVIDIA/feature/remove_cooker_condition
voegtlel Jun 5, 2025
7d0c8b2
Update data flow diagram
voegtlel Jun 5, 2025
f054214
Add cooking
voegtlel Jun 5, 2025
b916c90
Add cooking
voegtlel Jun 5, 2025
50169ac
Address review comments
philipp-fischer Jun 5, 2025
1595aaf
Merge commit 'be57d6cb52fdd35f51efd8327f94f6e5e959ec65' into feature/…
philipp-fischer Jun 5, 2025
667f463
Merge pull request #130 from NVIDIA/feature/samples_on_demand_docs
philipp-fischer Jun 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 0 additions & 16 deletions .github/actions/setupjust/action.yml
View file
Open in desktop

This file was deleted.

9 changes: 0 additions & 9 deletions .github/actions/setupjust/index.js
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion .github/workflows/documentation.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
uses: astral-sh/setup-uv@v5

- name: Install just
uses: ./.github/actions/setupjust
uses: extractions/setup-just@v3

- name: Install dependencies
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
uses: astral-sh/setup-uv@v5

- name: Install just
uses: ./.github/actions/setupjust
uses: extractions/setup-just@v3

- name: Install dependencies
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ruff.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
uses: astral-sh/setup-uv@v5

- name: Install just
uses: ./.github/actions/setupjust
uses: extractions/setup-just@v3

- name: Install dependencies
run: |
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,11 @@ jobs:
uses: astral-sh/setup-uv@v5

- name: Install just
uses: ./.github/actions/setupjust
uses: extractions/setup-just@v3

- name: Install minimum supported python version
run: |
uv python pin 3.9
uv python pin 3.10

- name: Install dependencies
run: |
Expand Down
212 changes: 0 additions & 212 deletions docs/source/advanced/advanced_dataformat.md
View file
Open in desktop

This file was deleted.

Loading