Refactoring for enhanced scheduler #22
Open
mlinmg wants to merge 42 commits into
Open
Conversation
…FasterTTS into enhanced_scheduler
…responsable for generating the input for the phonetic generation. The second is responsable for the input of the phonetic generation. Some more useful components are added, as a dynamic resource lock to maximize gpu utilization
mlinmg
force-pushed
the
enhanced_scheduler
branch
from
December 28, 2024 14:45
6cdd1ce to
e06e084
Compare
…d_scheduler # Conflicts: # src/auralis/__init__.py # src/auralis/common/definitions/dto/requests.py # src/auralis/common/definitions/protocols/openai.py # src/auralis/common/metrics/performance.py # src/auralis/common/utilities.py # src/auralis/common/vllm/hijack.py # src/auralis/core/tts.py # src/auralis/models/base.py # src/auralis/models/xttsv2/XTTSv2.py # tests/integration/memory_usage.py # tests/integration/test_genearte_audio_sync.py # tests/integration/test_generate_audio_async.py
…d_scheduler # Conflicts: # src/auralis/core/tts.py # src/auralis/models/xttsv2/utils/checkpoint_converter.py
Contributor
Author
|
@GiacomoLeoneMaria Can you check over the weekend to see where are the issues so we can merge this? |
|
Seems there is a conflict... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Preparing for the addition of new models the scheduler needs refactoring
current behavior:
The scheduler have two phases in the first one the model class prepare the async generators and the generation context ( the generation conditioning for instance and/or the voice cloining conditioning for the vocoder)
This work well with vllm but other models that do not use it might have problems
Desired Behavior
Ideally the scheduler should work like the vllm scheduler and add the request to the queue, and then dynamically batch them based on the system constraint, the scheduler should have three distinct phases:
The step needed for this taks are
get the concurrency valuesif the model will wold in a worst case scenario (avoid OOM during the process)process and store each queue results in buffer waiting to be passes to the next stepevery output is inserted in the next queueI've also introduced a memory manager class, but due to our limited bandwidth we can't introduce it by ourselfs, so any help is welcome!
I'll open an issue as a tracker for that implementation