HomeWise.AI Voice Assistant

HomeWise.AI is a voice driven assistant for daily household tasks. Using retrieval augmented generation(RAG) with real time audio input and output, grounded in user data with visible sources. Households switch between apps for schedules, reminders, and recipes. Text based assistants slow interaction and hide data sources.

Who this is for


Primary user: household member managing daily tasks.
 Secondary user: family member consuming shared schedules and reminders.

Table of contents

• Core Features
• High Level Flow
• Architecture Overview
• Key design choices and trade offs
• Key Risks Addressed
  • Security and privacy posture
  • Product impact
  • Project status
• Screenshot
• VoiceRAG Diagram
• Links- Usage
• Getting Started
• Contributing
• License
• Authors

Core Features

• Voice Interface: Interact with HomeWise.AI using voice capture via the browser's microphone.
• GPT Realtime API: Real time speech in and speech out.
• VoiceRAG (Retrieval-Augmented Generation): Enhanced data retrieval via the Azure AI Search to generation for accurate and relevant responses.
• Audio Autput: The response is retrieved from the GPT-4o Realtime API and played back to the user via the browser's speakers.
• Search Results: Visible search sources shown with each response.
• Personalized Recommendations: Tailored suggestions based on user preferences and past interactions.
• Privacy and Security: Robust measures to ensure user data is protected and private.
• Cross-Platform Compatibility: Accessible on various devices and platforms for convenience.
• Real-Time Updates: Continuously updated data and information for accurate and relevant responses.

High Level Flow

User presses to talk.
Audio streams to backend session.
Backend queries Azure AI Search for relevant data.
Model responds with grounded answer and audio.
UI plays audio and shows sources.

Architecture Overview

• Frontend: React, Tailwind.
• Backend: Node.js, Express.
• AI Services: Azure OpenAI GPT Realtime, Azure AI Search.

• Hosting: Azure Container Apps.
• Data: MongoDB for profiles and preferences, Blob Storage for documents.
• Observability: Azure Monitor and App Insights.

Key design choices and trade offs

• Voice interface over text only. Better usability accepted with higher privacy and cost risk.
• Realtime model over batch speech to text. Lower latency accepted with tighter session control needs.

• Persistent user memory over stateless sessions. Better personalisation is accepted with stronger access and retention requirements. 

• Azure AI Search over simple database queries. Higher retrieval quality accepted with index management work.

Key risks addressed


• Perceived always on microphone. Mitigated through push to talk UI and visible recording state.

• PII exposure in transcripts and logs. Mitigated through redaction and short retention.

• Prompt injection through retrieved content. Mitigated through allowlists and grounding rules.

• Unauthorised access to family data. Mitigated through role based access by household.

• Cost spikes from long voice sessions. Mitigated through session limits and alerts.

Security and privacy posture

User data treated as sensitive by default.
Explicit consent for stored preferences.
Clear separation between retrieval data and generation logic.
Design supports regulated consumer use cases.

Product impact


Reduces time to retrieve daily plans.
Improves trust through visible sources.
Increases engagement through voice interaction.

Project status


Working prototype focused on voice RAG and real time interaction.

• If you review this repo as a recruiter
Focus on the DESIGN.md for trade offs and the RISK_REGISTER.md for governance thinking.

Screenshot

VoiceRAG Diagram

Links

• Deployed Application
• GitHub Repo
• HomeWise.AI Presentation Slides

Getting Started

To get started with HomeWise.AI, follow these steps:

1. Clone the repository:

   git clone https://github.com/yourusername/HomeWise.AI.git

2. Navigate to the project directory:

   cd HomeWise.AI

3. Install the required dependencies:

   npm install 

   pip install -r requirements.txt```

4. Set Up Azure Credentials:

   • Create a new Azure account or sign in to your existing account and set up the necessary services.
   • Store your credentials in a .env file and load them into your application.

5. Run the Assistant

   python main.py

License

This project is licensed under the MIT License. See the LICENSE file for details.

Authors

Yuliya Tasheva

• For any questions, suggestions, or issues, please contact our team at

HomeWise.AI.com  ·  Email support@homewise.ai.com  ·  Copyright © 2024 All Rights Reserved. Site By Yuliya Tasheva®

Thank you for using and contributing to the HomeWise.AI website! Your support helps us continually improve and deliver the best voice assistance experience.