Chloi — Desktop Assistant

A privacy-first, highly intelligent Mac menu-bar AI companion, powered by Google Gemini.

Chloi is designed to act as your personal AI assistant — sitting quietly in your menu bar and assisting you on demand. It features deep screen awareness, stunning visual annotations, duplex voice conversations, and an autonomous background agent capable of executing actions on your Mac.

Features

• Advanced Screen Smartboard: Chloi can draw visual overlays directly on your screen to guide, debug, or teach you:
  • Glowing Underlines: Color-coded severity lines under text/code (red for errors, yellow for warnings, green for OK).
  • Numbered Callouts: Badges with leader lines pointing exactly to UI elements or text.
  • Visual HUD Steps: For math solutions or multi-step explanations, Chloi renders a clean, persistent left-aligned HUD column of step-by-step text.
  • Geometric Highlights: Glowing arrows, translucent bounding boxes, and custom shapes.
• Autonomous Background Agent: Say "Chloi agent, [task]" to spawn an asynchronous background researcher. The agent can:
  • Browse the web for live information.
  • Execute bash commands on your local system.
  • Read local files and configurations.
  • Execute AppleScript to control native macOS apps (Mail, Calendar, Finder).
  • Chloi will verbally announce the results to you once the background task is complete.
• Always-Visible Trust Indicator: A sleek floating pill indicator ensures you always know Chloi's state (Idle, Listening, Processing, Speaking). No covert listening.
• Duplex Voice Session: Press the hotkey to start a real-time conversation. Chloi can be interrupted naturally if you speak over it.
• Region Query: Drag-select a specific region of your screen to focus Chloi's attention exactly where you want it.
• Live Captions: Real-time subtitles of your conversation appear seamlessly on screen.

Prerequisites

• Node.js (v18 or newer recommended)
• A Gemini API Key from Google AI Studio

Setup

1. Install Dependencies:

   npm install

2. Build the Swift Native Overlay:

   cd src/native/overlay
   ./build.sh
   cd ../../..

3. Configure API Key:

   • Start the app.
   • Click the Chloi icon in your Mac menu bar.
   • Paste your Gemini API key in the settings panel.
   • (Alternatively, you can manually create a .env file in the root directory and add GEMINI_API_KEY=your_key_here).

Usage

Start the app in development mode:

npm start

Hotkeys

• Start/Stop Voice Session: Cmd + Option + Space
• Region Query (Drag to Select): Cmd + Option + Shift + Space

Example Voice Prompts

Use natural voice requests to leverage Chloi's intelligence:

• "Chloi, highlight the syntax errors in this code and explain them."
• "Chloi, solve this math equation." (Renders step-by-step HUD cards)
• "Chloi, what is happening on the right side of this image?"
• "Chloi agent, find the top 3 React state management libraries and summarize them." (Fires off the autonomous background agent)

Note: Visual markup clears automatically if you switch apps, click the screen, or after a short pause.

Build for Mac

To build a distributable .dmg file for macOS:

npm run build

The output will be generated in the dist folder.