Flora

A modern voice-enabled interactive platform with a PixiJS frontend and Python FastAPI backend that provides real-time voice activity detection and processing capabilities for rich multimedia experiences.

Overview

Flora combines beautiful visual elements with advanced voice processing technology. The platform features an animated starry background with smooth transitions between screens and responsive layouts that adapt to different device sizes.

Key Features

• Voice Activity Detection (VAD): Real-time speech recognition and processing
• Interactive Animations: Smooth, responsive animations with position interpolation
• Multimedia Integration: Audio playback and animated sprite management
• Responsive Design: Adapts to different screen sizes with proper scaling
• Extensible Plugin Architecture: Modular design with PIXI.js plugins

Project Structure

• client: Frontend application built with PixiJS 8, TypeScript, and VAD integration
• server: Backend service built with Python FastAPI

Core Technology

Frontend

• PIXI.js v8: High-performance WebGL rendering
• VAD Processing: WebAssembly-powered voice detection with confidence scoring
• TypeScript: For type-safe code
• Audio Processing: Real-time audio capture, background music, and sound effects
• Responsive Design: Adapts to different screen sizes with proper scaling

Backend

• Python FastAPI: Modern, high-performance web framework
• Docker: Containerization for easy deployment
• S3: Cloud storage for files

Getting Started

Client Setup

cd client
yarn install
yarn dev

Server Setup

cd server
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
mv app/.env.example app/.env
fastapi dev

The server will be available at http://localhost:8000 with API documentation at http://localhost:8000/docs

Project Structure

• /client/src/app - Application screens, UI components, and application logic
  • /screens - Main application screens and loading screens
  • /ui - Reusable UI components
  • /popups - Modal and overlay components
• /client/src/engine - Core engine functionality
  • /audio - Audio capture and playback management
  • /navigation - Screen navigation and transition system
  • /resize - Responsive design handling
  • /vad - Voice Activity Detection processing
• /server/app - Backend API services and endpoints

API Endpoints

Endpoint	Method	Description	Auth Required
/chat	POST	Process user queries and return responses	No
/login	POST	Admin authentication for JWT token	No
/upload	POST	Upload files to the system	Yes
/file	DELETE	Delete files from the system	Yes
/file	PATCH	Update file descriptions	Yes
/files	GET	Retrieve paginated list of files	Yes
/process	POST	Trigger file processing tasks	Yes

Architecture

The application uses a distributed architecture:

• FastAPI handles HTTP requests
• S3 stores uploaded files
• Client-Server communication for voice processing

Development

When developing server components, you can use the rebuild script to quickly rebuild and restart containers:

cd server
./rebuild.sh

Recent Updates

• Voice Activity Detection implementation
• Audio utilities refactoring
• Engine visibility handling improvements
• Enhanced server components with FastAPI endpoints and document processing