Added GHL Conversation Archive tool.

ghl-conversation-archive/README.md

@@ -0,0 +1,136 @@
+# GHL Conversation Archive
+
+Archives every conversation and message in a HighLevel sub-account to a folder on
+your machine. **Export only** — there is no import, for reasons set out below.
+
+Separate from the workflow backup extension on purpose: different data, different
+risk profile, different permissions.
+
+## Install
+
+1. Open `chrome://extensions`.
+2. Turn on **Developer mode** (top right).
+3. **Load unpacked** → select this folder.
+
+## Use
+
+1. Open a HighLevel sub-account tab and sign in as normal.
+2. Click the extension icon. It shows the sub-account and how many conversations
+   it holds.
+3. **Open the archiver…** — it runs in its own tab, because a full archive takes
+   far longer than a popup stays open.
+
+Three steps in that tab: grant durable access to this sub-account's origin, pick
+a local folder, run.
+
+```
+<folder>/
+├── manifest.json          counts, options, resume cursor, completed ids
+├── contacts.json          contactId -> name, email, phone
+├── conversations.jsonl    one conversation per line, metadata
+├── messages.jsonl         one message per line
+└── conversations/
+    └── <contact>-<id>.json
+```
+
+Measured on a real sub-account: 2,058 conversations at ~81 messages each,
+~2.4 KB per message — roughly 167,000 messages, 390–880 MB depending on which
+outputs you enable, about 20 minutes. The projection in the progress panel
+corrects itself from real bytes as it goes.
+
+The run is **resumable**. `manifest.json` holds the search cursor and every
+completed conversation id, so an interrupted run continues rather than starting
+over. Keep that file with the folder.
+
+For analysis, `messages.jsonl` loads directly:
+
+```sh
+duckdb -c "SELECT contactName, count(*) FROM read_json_auto('messages.jsonl') GROUP BY 1 ORDER BY 2 DESC LIMIT 20"
+```
+
+## How it works
+
+| Purpose | Request |
+|---|---|
+| Conversation list | `GET services.leadconnectorhq.com/conversations/search?locationId=&limit=&startAfterDate=` |
+| Messages in one thread | `GET .../conversations/{id}/messages?limit=&lastMessageId=` |
+| Contact record | `GET .../contacts/{id}` |
+
+**Paging.** Conversation search is cursor-paged, not offset-paged. `offset` and
+`page` are accepted and *silently ignored* — they return page one again. The
+cursor is the previous page's last conversation's `sort[0]`, passed back as
+`startAfterDate`. Messages page separately on `lastMessageId`.
+
+**The bulk endpoint.** `GET /conversations/messages/export` exists and is
+purpose-built for this, with proper cursor pagination. It rejects a browser
+session with `401 Can not fetch messages from non-OAuth channel`. It needs a
+Private Integration or OAuth token. If you make one, that endpoint is a better
+tool than this extension, and it would be a script rather than an extension.
+
+## Design notes
+
+**No credential handling.** The extension injects a function into the page's MAIN
+world and borrows `window.SHELL_STORE.$http` — the app's own axios instance, whose
+interceptor attaches the session token. The token is never read, copied, stored,
+or sent anywhere.
+
+**Permissions.** `activeTab` and `scripting`. Host access is declared *optional*
+and requested at runtime for one origin only, when you start an archive — a run
+that lasts twenty minutes needs access that outlives a popup click. Revoke it from
+the extension's details page whenever you like. No background service worker, no
+remote code, no network destination other than HighLevel's own API.
+
+**Written incrementally.** Files are written through the File System Access API as
+the run proceeds, so nothing large is held in memory and a crash costs you only
+the conversation in flight.
+
+## About the data
+
+This is customer PII: names, email addresses, phone numbers and full message
+bodies, written unencrypted to the folder you pick. Keep it out of any git
+repository — not because a private repo is insecure, but because git history is
+permanent and this is exactly the data a deletion request applies to.
+
+## Why there is no import
+
+Export is solved. Import is not, and the blocker is not contact IDs.
+
+Mapping old contact ids to new ones is the easy part, which is why `contacts.json`
+is written: id, name, email and phone, enough to match contacts in a target
+account by email or normalised phone.
+
+The hard parts:
+
+- **No restore API.** Nothing recreates a conversation as it was. The closest are
+  *Add Inbound Message* and *Add External Outbound Call*, which record messages
+  one at a time onto a contact.
+- **Backdating is unverified.** Whether those endpoints accept a custom timestamp
+  is not stated consistently in the public documentation, and it is not something
+  to establish by writing to a live account. If they do not, every restored
+  message lands with today's date and the history is worthless as history.
+- **Writing messages fires automations.** Inbound messages are a trigger type.
+  Replaying ~167,000 of them into an account with live workflows could start
+  automations en masse, including outbound sends to real customers. This is the
+  genuine hazard, and it is worse than data loss.
+- **Fidelity is lost anyway.** Call recordings, transcriptions, email threading
+  headers, delivery status, attachments and read state do not round-trip.
+- **Rate limits.** At roughly 100 requests per 10 seconds, 167,000 writes is
+  measured in days.
+
+Treat conversation history as an archive, not a restorable backup. If you migrate
+accounts, carry a summary onto the contact — a note, or a link into this archive —
+rather than replaying the messages.
+
+## Limits
+
+- One sub-account per run — whichever the tab is on.
+- Message bodies come back inline but also carry a `bodyStorageUrl`; very long
+  emails are truncated in the API response and the full body lives at that URL.
+  This archives what the API returns and does not chase those URLs.
+- These are undocumented internal endpoints. They can change without notice.
+
+## Changelog
+
+### 1.0.0
+- Split out of the workflow backup extension, where it never belonged.
+- Resumable archive of all conversations and messages to a local folder.

ghl-conversation-archive/agent.js

@@ -0,0 +1,43 @@
+// Injected into the HighLevel page's MAIN world by chrome.scripting.executeScript.
+// Serialized and re-parsed there, so this must be entirely self-contained: no
+// imports, no closure variables.
+//
+// Why the MAIN world: the app keeps its authenticated HTTP client on
+// window.SHELL_STORE.$http, whose interceptor attaches the session token. By
+// borrowing that client we never read, store, or transmit a credential
+// ourselves -- requests go out exactly as the app's own requests do.
+
+/** Confirms we are on a HighLevel app page and reports the location in view. */
+export function probePage() {
+  const findJwt = (node, depth) => {
+    if (!node || depth > 4) return null;
+    if (typeof node === 'string') {
+      return /^ey[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\./.test(node) ? node : null;
+    }
+    if (typeof node !== 'object') return null;
+    for (const key of Object.keys(node)) {
+      const hit = findJwt(node[key], depth + 1);
+      if (hit) return hit;
+    }
+    return null;
+  };
+
+  const store = window.SHELL_STORE;
+  if (!store) return { ok: false, reason: 'not-highlevel' };
+
+  const fromUrl = location.pathname.match(/\/location\/([A-Za-z0-9]+)/);
+  let locationId = fromUrl ? fromUrl[1] : null;
+  let locationName = null;
+  try {
+    const current = store.state.locations.currentLocation;
+    locationId = locationId || current.id || current._id;
+    locationName = current.name || null;
+  } catch (e) { /* location name is cosmetic */ }
+
+  if (!locationId) return { ok: false, reason: 'no-location' };
+
+  const mode = store.$http ? 'client' : (findJwt(store.state && store.state.auth, 0) ? 'token' : null);
+  if (!mode) return { ok: false, reason: 'no-auth' };
+
+  return { ok: true, locationId, locationName, mode, origin: location.origin };
+}

ghl-conversation-archive/archive.css

@@ -0,0 +1,62 @@
+:root {
+  color-scheme: light dark;
+  --bg: #ffffff; --fg: #16181d; --muted: #6b7280; --line: #e4e7ec; --sunk: #f6f7f9;
+  --accent: #2f6fed; --accent-fg: #fff; --ok: #167c46; --err: #b42318;
+}
+@media (prefers-color-scheme: dark) {
+  :root {
+    --bg: #16181d; --fg: #f2f4f7; --muted: #98a2b3; --line: #2c3038; --sunk: #1c1f26;
+    --accent: #5b8dff; --accent-fg: #0b0d11; --ok: #5fd39b; --err: #ff9a92;
+  }
+}
+* { box-sizing: border-box; }
+body {
+  margin: 0; padding: 32px 20px; background: var(--bg); color: var(--fg);
+  font: 14px/1.6 system-ui, -apple-system, "Segoe UI", sans-serif;
+}
+main { max-width: 660px; margin: 0 auto; }
+h1 { margin: 0 0 2px; font-size: 20px; }
+h2 { margin: 0 0 4px; font-size: 14px; }
+p { margin: 4px 0 0; }
+code { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12.5px; background: var(--sunk); padding: 1px 4px; border-radius: 4px; }
+.muted { color: var(--muted); font-size: 13px; }
+em.muted { display: block; font-style: normal; margin-top: 1px; }
+
+.steps { list-style: none; margin: 24px 0 0; padding: 0; }
+.step { padding: 16px 0; border-top: 1px solid var(--line); }
+.step.is-off { opacity: .4; pointer-events: none; }
+.step.is-done h2::after { content: " ✓"; color: var(--ok); }
+
+button {
+  margin-top: 10px; padding: 8px 14px; border: 1px solid var(--line); border-radius: 7px;
+  background: transparent; color: var(--fg); font: inherit; cursor: pointer;
+}
+button.primary { border: 0; background: var(--accent); color: var(--accent-fg); font-weight: 600; }
+button:disabled { opacity: .5; cursor: default; }
+.controls { display: flex; gap: 8px; }
+
+.check { display: flex; gap: 8px; margin-top: 12px; cursor: pointer; }
+.check input { margin: 4px 0 0; flex: none; }
+
+.state { font-size: 13px; min-height: 18px; }
+.state.ok { color: var(--ok); }
+.state.err { color: var(--err); }
+
+.stats { display: grid; grid-template-columns: repeat(4, 1fr); gap: 8px; margin-top: 14px; }
+.stats-5 { grid-template-columns: repeat(5, 1fr); }
+.opts { margin: 14px 0 0; padding: 10px 12px; border: 1px solid var(--line); border-radius: 8px; }
+.opts legend { padding: 0 4px; }
+.opts .check { margin-top: 8px; }
+.opts .check:first-of-type { margin-top: 2px; }
+.stats div { padding: 10px; border-radius: 8px; background: var(--sunk); text-align: center; }
+.stats b { display: block; font-size: 18px; font-variant-numeric: tabular-nums; }
+.stats span { font-size: 11px; }
+
+.bar { height: 5px; margin-top: 12px; border-radius: 3px; background: var(--line); overflow: hidden; }
+.bar span { display: block; height: 100%; width: 0; background: var(--accent); transition: width .3s ease; }
+
+details { margin-top: 12px; font-size: 13px; }
+details ul { margin: 6px 0 0; padding-left: 18px; }
+.note { margin-top: 28px; padding-top: 16px; border-top: 1px solid var(--line); }
+.note ul { padding-left: 18px; }
+.note li { margin-bottom: 3px; }

ghl-conversation-archive/archive.html

@@ -0,0 +1,107 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Conversation archive</title>
+<link rel="stylesheet" href="archive.css">
+</head>
+<body>
+<main>
+  <h1>Conversation archive</h1>
+  <p id="sub" class="muted">Connecting to the HighLevel tab&hellip;</p>
+
+  <ol class="steps">
+    <li id="s1" class="step">
+      <h2>1 · Grant access to this sub-account</h2>
+      <p class="muted">The archive runs for a long time, so it needs durable permission for this one
+        origin rather than the per-click access the popup uses. You can revoke it at any time from
+        the extension's details page.</p>
+      <button id="grant" class="primary">Grant access</button>
+      <p id="grantState" class="state"></p>
+    </li>
+
+    <li id="s2" class="step is-off">
+      <h2>2 · Choose where to write it</h2>
+      <p class="muted">Pick an empty folder on this machine. Files are written as the run goes, so
+        nothing large is held in memory and a crash costs you only the conversation in flight.</p>
+      <button id="pick" class="primary">Choose folder&hellip;</button>
+      <p id="pickState" class="state"></p>
+      <fieldset class="opts">
+        <legend class="muted">What to write</legend>
+        <label class="check">
+          <input id="optFiles" type="checkbox" checked>
+          <span>One file per conversation
+            <em class="muted">Readable threads you can open individually. Best for the archive and
+            dispute-answering case.</em></span>
+        </label>
+        <label class="check">
+          <input id="optJsonl" type="checkbox" checked>
+          <span><code>messages.jsonl</code>
+            <em class="muted">One message per line. This is the one to stream into SQLite or DuckDB
+            for search and analysis.</em></span>
+        </label>
+        <label class="check">
+          <input id="optPretty" type="checkbox">
+          <span>Pretty-print the per-conversation files
+            <em class="muted">Easier to read by eye, roughly a quarter larger.</em></span>
+        </label>
+      </fieldset>
+
+      <label class="check">
+        <input id="withContacts" type="checkbox" checked>
+        <span>Also fetch each contact record.
+          <em class="muted">Adds phone numbers, which the conversation list does not return. Needed if
+          you ever want to map these contacts into another account. Costs roughly one extra request
+          per contact.</em></span>
+      </label>
+    </li>
+
+    <li id="s3" class="step is-off">
+      <h2>3 · Run</h2>
+      <div class="controls">
+        <button id="start" class="primary">Start</button>
+        <button id="pause" hidden>Pause</button>
+        <button id="stop" hidden>Stop</button>
+      </div>
+
+      <div id="stats" class="stats stats-5" hidden>
+        <div><b id="nConv">0</b><span class="muted">conversations</span></div>
+        <div><b id="nMsg">0</b><span class="muted">messages</span></div>
+        <div><b id="nMb">0</b><span class="muted">MB written</span></div>
+        <div><b id="eta">—</b><span class="muted">remaining</span></div>
+        <div><b id="proj">—</b><span class="muted">projected total</span></div>
+      </div>
+      <div class="bar" hidden id="barWrap"><span id="bar"></span></div>
+      <p id="now" class="muted"></p>
+      <p id="runState" class="state"></p>
+      <details id="errBox" hidden>
+        <summary><span id="errCount">0</span> conversation(s) failed</summary>
+        <ul id="errList"></ul>
+      </details>
+    </li>
+  </ol>
+
+  <section class="note">
+    <h2>What lands in the folder</h2>
+    <ul>
+      <li><code>conversations/&lt;contact&gt;-&lt;id&gt;.json</code> — one readable file per thread.</li>
+      <li><code>messages.jsonl</code> — one message per line. Stream this into SQLite or DuckDB.</li>
+      <li><code>conversations.jsonl</code> — one conversation per line, metadata only.</li>
+      <li><code>contacts.json</code> — contact id to name, email and phone. This is the mapping table
+        a future migration would join on.</li>
+      <li><code>manifest.json</code> — counts and the resume cursor. Keep it; it is what lets an
+        interrupted run pick up where it stopped.</li>
+    </ul>
+    <p class="muted">Measured on a 24-conversation sample of this sub-account: about 81 messages per
+      conversation at roughly 2.4&nbsp;KB each. Both outputs together land near 880&nbsp;MB for 2,058
+      conversations; either one alone is closer to 390&nbsp;MB. The projected figure above corrects
+      itself from real bytes as the run proceeds.</p>
+    <p class="muted">This is customer data: names, email addresses, phone numbers and full message
+      bodies. It is written unencrypted, as you asked. Keep it out of any git repository — not
+      because a private repo is insecure, but because git history is permanent and this is exactly
+      the data a deletion request applies to.</p>
+  </section>
+</main>
+<script type="module" src="archive.js"></script>
+</body>
+</html>