Feature Request: Speech based Controls

[rajeshgayathri2003]

Hi @kaushav07 !

I see that while speech based control is one of the core features of this project, no one has taken it up yet. I would like to propose the following workflow:

1. Speech-to-text algorithm to convert the request of the user to text
2. Use an image captioning algorithm like CLIP or DinoV2 to help complete the user's request based on visual input.

Would be happy to take this up!

[kaushav07]

Sounds like a great addition, @rajeshgayathri2003 Speech-based control is indeed a crucial feature, and your suggested workflow aligns well with our project’s direction. Let’s discuss the technical specifics so we can plan the implementation effectively!

[rajeshgayathri2003]

Thanks @kaushav07 !

I plan on proceeding this way -

1. Use the speech_recognition and pyttsx3 to convert the user's speech into text.
2. To effectively, execute the user's instructions, I suggest the use of an image captioning model like BLIP, which can be easily used via HuggingFace transformers. This can help identify various parts of the image and help the user accordingly
3. The output is given in speech format via pyttsx3

If a more specific workflow is required, please let me know what is expected.

Thanks!

[kaushav07]

Good approach @rajeshgayathri2003 , proceed with this plan and we can refine the workflow together if any adjustments are needed during implementation!

[rajeshgayathri2003]

@kaushav07 I will get started on the same and keep you posted!

[shreeraksha2112]

@rajeshgayathri2003 this issue is assigned to you