Personal Assistant - "Ask My Life"

A personal AI assistant that answers questions about my own life, in plain English.

"Who did I text the most in 2015?" · "Where was I on March 10, 2019?" · "How did I sleep last week?"

Your question goes to Claude, which decides which local tools to call - full-text search over 15 years of my text messages, my location history, my Oura Ring health data - and comes back with specifics: dates, names, quotes, numbers.

Why this exists

More than a decade of my life is sitting in data I technically "have" but can't actually use. Fifteen years of text messages across three phones and two carriers. Tens of thousands of photos, most of them geotagged. Years of sleep and activity data from a ring on my finger. Every one of those systems can store my history - none of them can answer a question about it.

The questions I actually have are small and constant:

• "What's Alex's address again?" - I never saved it to Contacts, but Alex texted it to me once in 2019. It's in there. Before this project, "in there" meant scrolling for ten minutes and giving up.
• "When's Mom's birthday dinner?" - planned over a text thread three weeks ago, buried under everything since.
• "What was that restaurant we went to in Austin?" - someone texted the name, or I took a geotagged photo out front. Either way the answer exists; I just couldn't reach it.
• "Where was I the weekend of March 10, 2019?" - my photos and location history know exactly, down to the hour.

The search built into Messages or Photos assumes you remember a keyword. I almost never do - what I remember is the shape of the memory (roughly when, roughly who, roughly what), and that's exactly what an LLM with tools is good at turning into a real query. So the answer to "make my own history searchable" turned out to be: consolidate everything into local databases, and put an agent in front of them.

About the builder: I'm a sales professional, not a software engineer - and I'm very upfront about that 🙂. I built this with AI pair-programming tools to solve a real problem: my own history was scattered across a decade of phone backups, exports, and apps, and nothing could answer a simple question about it. This repo is part of my portfolio - it shows how far a motivated non-programmer can get by pairing domain thinking with modern AI tooling.

---

The extensibility story: a feature = one file

The core design decision: the agent (agent.py) and the web server (app.py) never change. Every capability is a single self-describing file in tools/, auto-discovered at startup.

personal-assistant/
├── agent.py          generic Claude loop - auto-discovers tools, routes, answers
├── app.py            web server: chat UI (/) + JSON endpoint (POST /ask)
├── mcp_server.py     exposes the same tools to the Claude desktop/mobile app (MCP)
├── tools/            EVERY capability lives here (auto-discovered)
│   ├── search_texts.py   full-text search over the consolidated message archive
│   ├── text_stats.py     counts & rankings ("who did I text most in 2015?")
│   ├── where_was_i.py    location on any date (timeline + geotagged photos)
│   ├── oura.py           sleep / readiness / activity / heart-rate queries
│   └── _template.py      copy this to add a new tool
├── ingest/           scripts that build & refresh the local databases
├── web/              the chat page (installs as an iPhone home-screen PWA)
├── config/           .env.example - copy to .env and add your own keys
└── data/             (NOT in the repo) the local databases live here

Adding a capability:

1. cp tools/_template.py tools/<your_tool>.py
2. Fill in TOOL (name, a description of when Claude should use it, input schema) and run(args).
3. Restart. The agent auto-discovers it and Claude can now call it. No other file changes.

The same registry powers three front-ends: the web/PWA chat page, an iOS Shortcut/Siri (anything that can POST to /ask), and an MCP server for the Claude desktop and mobile apps.

The mobile solution: getting this onto my phone

Here's the thing - almost none of these questions come up while I'm sitting at a desk. "What's Alex's address?" matters in the car. So the project only really pays off if it lives on my phone, and I wanted that without building a native app (I'm not learning Swift for this). Two pieces solve it:

1. The PWA chat page (web/index.html). One self-contained HTML file, served by Flask at /. Open it in Safari, tap Share → Add to Home Screen, and it launches full-screen with its own icon - which makes it feel like a native app even though the entire "app" is a single file. And it's a real chat app, not a toy:

• a slide-in drawer of saved conversations (kept in the browser's localStorage, so chats survive between uses)
• a model picker - Sonnet (fast), Opus (smartest), Haiku (cheapest)
• a running cost readout per chat, so I always know what a question cost me (usually a fraction of a cent)

Under the hood it's deliberately simple: every message is a POST /ask to the Flask JSON API in app.py, carrying the question, the last few turns of history, and the chosen model. The server runs the agent loop and returns the answer plus token and cost numbers. Anything that can make that POST is a front-end - which is exactly how the iOS Shortcut / Siri integration works too.

2. The MCP server (mcp_server.py). The same tools/ registry, exposed over the Model Context Protocol - so instead of my app calling the model, an AI client can call my tools. In stdio mode it plugs straight into Claude Desktop; set TRANSPORT=http and it becomes a streamable HTTP server that the AI app on my phone connects to as a remote connector. Remote requests are gated by an Authorization: Bearer <token> check (set MCP_TOKEN), so however you choose to expose the endpoint, only authorized clients get in.

Same brain, two doors: the PWA when I want my own chat interface, MCP when I want my everyday AI assistant to reach into my history from anywhere.

Privacy by design

This project exists to query deeply personal data - so the repo is built to ship zero data:

• All data stays local. Messages, location history, and health data live in SQLite files under data/, on my own machines. Nothing is uploaded anywhere except the minimal tool results sent to the Claude API to answer each question.
• The repo contains code only. A default-deny .gitignore excludes everything at the root and re-includes only code and docs. Databases, raw exports, logs, and secrets cannot be committed, even by accident.
• Secrets live in config/.env (never committed - config/.env.example is the template).
• The ingest scripts read from sources on the local machine (macOS Messages database, Photos library, phone-backup XML exports, the Oura API) and write only to local data/ files.

Sample interactions

Illustrative examples with fictional data - no real names, messages, or locations appear in this repository.

Q: What's Alex's address? I never saved it.

A: Alex texted it to you on 2019-08-11: "we're at 412 Maple Ct now, come by whenever" - that's the most recent address Alex has sent.

Q: When did Jordan mention the cabin trip?

A: Jordan first brought up the cabin trip on 2021-06-14 ("we should do the cabin thing again in August") and you two settled on the weekend of Aug 20-22 in messages on 2021-07-02.

Q: Where was I on October 3, 2022?

A: Geotagged photos put you in Portland, OR that afternoon (3 photos near the waterfront, 2:10-4:45 PM), and that evening you texted Sam "just landed back home" - so: day trip to Portland, home by night.

Q: How did I sleep this week compared to my average?

A: You averaged 7h 12m across the last 7 nights (efficiency 91%), about 20 minutes above your 90-day average. Best night was Tuesday (8h 05m); Friday was the outlier at 5h 40m.

The fun side

Once your history lives in queryable databases, it stops being just a lookup service and becomes raw material. Some of what this architecture makes possible - each one is just another file in tools/ (or a script over the same data):

• A map of everywhere I've been. Every geotagged photo is a timestamped coordinate. Plot all of them and you get a personal travel map built from evidence, not memory - including the trips you'd half-forgotten you took.
• Where I've spent my time, year by year. Cluster the location history by year and city: which places dominated each era of my life, when a move actually happened (versus when the lease said it did), which year I traveled the most.
• Texting-pattern stats. text_stats.py already answers "who did I text most in 2015?" - the same data supports friendship timelines (who entered and faded from my life, by message volume per year), busiest texting hours, average reply times, and how all of that shifted across a decade.
• Cross-source questions. The interesting queries combine sources: "how do I sleep after nights I'm out past 11?" (location + Oura), or "show me a photo from the day Jordan and I planned the cabin trip" (texts + photos).

None of this requires touching the agent - the data is already there, and a new question is a new tool file.

Setup

Requires Python 3.12+ and an Anthropic API key. You bring your own data - the ingest scripts build the local databases from your exports; the repo ships none.

git clone <this repo> && cd personal-assistant
python3.12 -m venv venv && venv/bin/pip install -r requirements.txt
cp config/.env.example config/.env    # then put your real ANTHROPIC_API_KEY in config/.env
venv/bin/python app.py                # serves on http://0.0.0.0:8765

Open http://<host>:8765 - or POST {"question": "..."} to /ask.

Data ingestion (each source is optional)

• Texts - ingest/sync_imessage.py incrementally pulls new iMessages from the Mac's chat.db (needs Full Disk Access) into data/all_texts.sqlite; historical SMS / Google Voice archives were merged in from standard XML backup exports.
• Locations - ingest/extract_photo_locations.py distills geotagged-photo coordinates from the macOS Photos library; a Google Timeline export supplies data/location-history.json.
• Oura Ring - ingest/oura_auth.py does a one-time OAuth2 browser authorization (create an app at cloud.ouraring.com, set redirect URI http://localhost:8123/callback, put the client id/secret in config/.env); after that ingest/sync_oura.py refreshes tokens and syncs headlessly forever.
• Automation - update.sh chains the syncs and pushes to an always-on home server; ingest/com.eric.pa-sync.plist schedules it daily via launchd.

Front-ends (all hit POST /ask)

• Web / PWA: open in Safari → Share → Add to Home Screen.
• iOS Shortcut / Siri: a Shortcut that prompts for text, POSTs to /ask, shows the answer.
• Claude app (MCP): python mcp_server.py for Claude Desktop, or TRANSPORT=http behind a bearer token for the mobile app.
• Reachable from the phone anywhere over an encrypted private network.