diff --git a/ghl-conversation-archive/README.md b/ghl-conversation-archive/README.md new file mode 100644 index 0000000..3da7fd2 --- /dev/null +++ b/ghl-conversation-archive/README.md @@ -0,0 +1,136 @@ +# GHL Conversation Archive + +Archives every conversation and message in a HighLevel sub-account to a folder on +your machine. **Export only** — there is no import, for reasons set out below. + +Separate from the workflow backup extension on purpose: different data, different +risk profile, different permissions. + +## Install + +1. Open `chrome://extensions`. +2. Turn on **Developer mode** (top right). +3. **Load unpacked** → select this folder. + +## Use + +1. Open a HighLevel sub-account tab and sign in as normal. +2. Click the extension icon. It shows the sub-account and how many conversations + it holds. +3. **Open the archiver…** — it runs in its own tab, because a full archive takes + far longer than a popup stays open. + +Three steps in that tab: grant durable access to this sub-account's origin, pick +a local folder, run. + +``` +/ +├── manifest.json counts, options, resume cursor, completed ids +├── contacts.json contactId -> name, email, phone +├── conversations.jsonl one conversation per line, metadata +├── messages.jsonl one message per line +└── conversations/ + └── -.json +``` + +Measured on a real sub-account: 2,058 conversations at ~81 messages each, +~2.4 KB per message — roughly 167,000 messages, 390–880 MB depending on which +outputs you enable, about 20 minutes. The projection in the progress panel +corrects itself from real bytes as it goes. + +The run is **resumable**. `manifest.json` holds the search cursor and every +completed conversation id, so an interrupted run continues rather than starting +over. Keep that file with the folder. + +For analysis, `messages.jsonl` loads directly: + +```sh +duckdb -c "SELECT contactName, count(*) FROM read_json_auto('messages.jsonl') GROUP BY 1 ORDER BY 2 DESC LIMIT 20" +``` + +## How it works + +| Purpose | Request | +|---|---| +| Conversation list | `GET services.leadconnectorhq.com/conversations/search?locationId=&limit=&startAfterDate=` | +| Messages in one thread | `GET .../conversations/{id}/messages?limit=&lastMessageId=` | +| Contact record | `GET .../contacts/{id}` | + +**Paging.** Conversation search is cursor-paged, not offset-paged. `offset` and +`page` are accepted and *silently ignored* — they return page one again. The +cursor is the previous page's last conversation's `sort[0]`, passed back as +`startAfterDate`. Messages page separately on `lastMessageId`. + +**The bulk endpoint.** `GET /conversations/messages/export` exists and is +purpose-built for this, with proper cursor pagination. It rejects a browser +session with `401 Can not fetch messages from non-OAuth channel`. It needs a +Private Integration or OAuth token. If you make one, that endpoint is a better +tool than this extension, and it would be a script rather than an extension. + +## Design notes + +**No credential handling.** The extension injects a function into the page's MAIN +world and borrows `window.SHELL_STORE.$http` — the app's own axios instance, whose +interceptor attaches the session token. The token is never read, copied, stored, +or sent anywhere. + +**Permissions.** `activeTab` and `scripting`. Host access is declared *optional* +and requested at runtime for one origin only, when you start an archive — a run +that lasts twenty minutes needs access that outlives a popup click. Revoke it from +the extension's details page whenever you like. No background service worker, no +remote code, no network destination other than HighLevel's own API. + +**Written incrementally.** Files are written through the File System Access API as +the run proceeds, so nothing large is held in memory and a crash costs you only +the conversation in flight. + +## About the data + +This is customer PII: names, email addresses, phone numbers and full message +bodies, written unencrypted to the folder you pick. Keep it out of any git +repository — not because a private repo is insecure, but because git history is +permanent and this is exactly the data a deletion request applies to. + +## Why there is no import + +Export is solved. Import is not, and the blocker is not contact IDs. + +Mapping old contact ids to new ones is the easy part, which is why `contacts.json` +is written: id, name, email and phone, enough to match contacts in a target +account by email or normalised phone. + +The hard parts: + +- **No restore API.** Nothing recreates a conversation as it was. The closest are + *Add Inbound Message* and *Add External Outbound Call*, which record messages + one at a time onto a contact. +- **Backdating is unverified.** Whether those endpoints accept a custom timestamp + is not stated consistently in the public documentation, and it is not something + to establish by writing to a live account. If they do not, every restored + message lands with today's date and the history is worthless as history. +- **Writing messages fires automations.** Inbound messages are a trigger type. + Replaying ~167,000 of them into an account with live workflows could start + automations en masse, including outbound sends to real customers. This is the + genuine hazard, and it is worse than data loss. +- **Fidelity is lost anyway.** Call recordings, transcriptions, email threading + headers, delivery status, attachments and read state do not round-trip. +- **Rate limits.** At roughly 100 requests per 10 seconds, 167,000 writes is + measured in days. + +Treat conversation history as an archive, not a restorable backup. If you migrate +accounts, carry a summary onto the contact — a note, or a link into this archive — +rather than replaying the messages. + +## Limits + +- One sub-account per run — whichever the tab is on. +- Message bodies come back inline but also carry a `bodyStorageUrl`; very long + emails are truncated in the API response and the full body lives at that URL. + This archives what the API returns and does not chase those URLs. +- These are undocumented internal endpoints. They can change without notice. + +## Changelog + +### 1.0.0 +- Split out of the workflow backup extension, where it never belonged. +- Resumable archive of all conversations and messages to a local folder. diff --git a/ghl-conversation-archive/agent.js b/ghl-conversation-archive/agent.js new file mode 100644 index 0000000..fe7c142 --- /dev/null +++ b/ghl-conversation-archive/agent.js @@ -0,0 +1,43 @@ +// Injected into the HighLevel page's MAIN world by chrome.scripting.executeScript. +// Serialized and re-parsed there, so this must be entirely self-contained: no +// imports, no closure variables. +// +// Why the MAIN world: the app keeps its authenticated HTTP client on +// window.SHELL_STORE.$http, whose interceptor attaches the session token. By +// borrowing that client we never read, store, or transmit a credential +// ourselves -- requests go out exactly as the app's own requests do. + +/** Confirms we are on a HighLevel app page and reports the location in view. */ +export function probePage() { + const findJwt = (node, depth) => { + if (!node || depth > 4) return null; + if (typeof node === 'string') { + return /^ey[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\./.test(node) ? node : null; + } + if (typeof node !== 'object') return null; + for (const key of Object.keys(node)) { + const hit = findJwt(node[key], depth + 1); + if (hit) return hit; + } + return null; + }; + + const store = window.SHELL_STORE; + if (!store) return { ok: false, reason: 'not-highlevel' }; + + const fromUrl = location.pathname.match(/\/location\/([A-Za-z0-9]+)/); + let locationId = fromUrl ? fromUrl[1] : null; + let locationName = null; + try { + const current = store.state.locations.currentLocation; + locationId = locationId || current.id || current._id; + locationName = current.name || null; + } catch (e) { /* location name is cosmetic */ } + + if (!locationId) return { ok: false, reason: 'no-location' }; + + const mode = store.$http ? 'client' : (findJwt(store.state && store.state.auth, 0) ? 'token' : null); + if (!mode) return { ok: false, reason: 'no-auth' }; + + return { ok: true, locationId, locationName, mode, origin: location.origin }; +} diff --git a/ghl-conversation-archive/archive.css b/ghl-conversation-archive/archive.css new file mode 100644 index 0000000..26505ab --- /dev/null +++ b/ghl-conversation-archive/archive.css @@ -0,0 +1,62 @@ +:root { + color-scheme: light dark; + --bg: #ffffff; --fg: #16181d; --muted: #6b7280; --line: #e4e7ec; --sunk: #f6f7f9; + --accent: #2f6fed; --accent-fg: #fff; --ok: #167c46; --err: #b42318; +} +@media (prefers-color-scheme: dark) { + :root { + --bg: #16181d; --fg: #f2f4f7; --muted: #98a2b3; --line: #2c3038; --sunk: #1c1f26; + --accent: #5b8dff; --accent-fg: #0b0d11; --ok: #5fd39b; --err: #ff9a92; + } +} +* { box-sizing: border-box; } +body { + margin: 0; padding: 32px 20px; background: var(--bg); color: var(--fg); + font: 14px/1.6 system-ui, -apple-system, "Segoe UI", sans-serif; +} +main { max-width: 660px; margin: 0 auto; } +h1 { margin: 0 0 2px; font-size: 20px; } +h2 { margin: 0 0 4px; font-size: 14px; } +p { margin: 4px 0 0; } +code { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12.5px; background: var(--sunk); padding: 1px 4px; border-radius: 4px; } +.muted { color: var(--muted); font-size: 13px; } +em.muted { display: block; font-style: normal; margin-top: 1px; } + +.steps { list-style: none; margin: 24px 0 0; padding: 0; } +.step { padding: 16px 0; border-top: 1px solid var(--line); } +.step.is-off { opacity: .4; pointer-events: none; } +.step.is-done h2::after { content: " ✓"; color: var(--ok); } + +button { + margin-top: 10px; padding: 8px 14px; border: 1px solid var(--line); border-radius: 7px; + background: transparent; color: var(--fg); font: inherit; cursor: pointer; +} +button.primary { border: 0; background: var(--accent); color: var(--accent-fg); font-weight: 600; } +button:disabled { opacity: .5; cursor: default; } +.controls { display: flex; gap: 8px; } + +.check { display: flex; gap: 8px; margin-top: 12px; cursor: pointer; } +.check input { margin: 4px 0 0; flex: none; } + +.state { font-size: 13px; min-height: 18px; } +.state.ok { color: var(--ok); } +.state.err { color: var(--err); } + +.stats { display: grid; grid-template-columns: repeat(4, 1fr); gap: 8px; margin-top: 14px; } +.stats-5 { grid-template-columns: repeat(5, 1fr); } +.opts { margin: 14px 0 0; padding: 10px 12px; border: 1px solid var(--line); border-radius: 8px; } +.opts legend { padding: 0 4px; } +.opts .check { margin-top: 8px; } +.opts .check:first-of-type { margin-top: 2px; } +.stats div { padding: 10px; border-radius: 8px; background: var(--sunk); text-align: center; } +.stats b { display: block; font-size: 18px; font-variant-numeric: tabular-nums; } +.stats span { font-size: 11px; } + +.bar { height: 5px; margin-top: 12px; border-radius: 3px; background: var(--line); overflow: hidden; } +.bar span { display: block; height: 100%; width: 0; background: var(--accent); transition: width .3s ease; } + +details { margin-top: 12px; font-size: 13px; } +details ul { margin: 6px 0 0; padding-left: 18px; } +.note { margin-top: 28px; padding-top: 16px; border-top: 1px solid var(--line); } +.note ul { padding-left: 18px; } +.note li { margin-bottom: 3px; } diff --git a/ghl-conversation-archive/archive.html b/ghl-conversation-archive/archive.html new file mode 100644 index 0000000..d1aaa87 --- /dev/null +++ b/ghl-conversation-archive/archive.html @@ -0,0 +1,107 @@ + + + + +Conversation archive + + + +
+

Conversation archive

+

Connecting to the HighLevel tab…

+ +
    +
  1. +

    1 · Grant access to this sub-account

    +

    The archive runs for a long time, so it needs durable permission for this one + origin rather than the per-click access the popup uses. You can revoke it at any time from + the extension's details page.

    + +

    +
    2. + +
  2. +

    2 · Choose where to write it

    +

    Pick an empty folder on this machine. Files are written as the run goes, so + nothing large is held in memory and a crash costs you only the conversation in flight.

    + +

    +
    + What to write + + + +
    + + +
    3. + +
  3. +

    3 · Run

    +
    + + + +
    + + + +

    +

    + +
    4. +
+ +
+

What lands in the folder

+
    +
  • conversations/<contact>-<id>.json — one readable file per thread.
    • +
  • messages.jsonl — one message per line. Stream this into SQLite or DuckDB.
    • +
  • conversations.jsonl — one conversation per line, metadata only.
    • +
  • contacts.json — contact id to name, email and phone. This is the mapping table + a future migration would join on.
    • +
  • manifest.json — counts and the resume cursor. Keep it; it is what lets an + interrupted run pick up where it stopped.
    • +
+

Measured on a 24-conversation sample of this sub-account: about 81 messages per + conversation at roughly 2.4 KB each. Both outputs together land near 880 MB for 2,058 + conversations; either one alone is closer to 390 MB. The projected figure above corrects + itself from real bytes as the run proceeds.

+

This is customer data: names, email addresses, phone numbers and full message + bodies. It is written unencrypted, as you asked. Keep it out of any git repository — not + because a private repo is insecure, but because git history is permanent and this is exactly + the data a deletion request applies to.

+
+
+ + + diff --git a/ghl-conversation-archive/archive.js b/ghl-conversation-archive/archive.js new file mode 100644 index 0000000..29fe848 --- /dev/null +++ b/ghl-conversation-archive/archive.js @@ -0,0 +1,399 @@ +import { probePage } from './agent.js'; +import { fetchConversationsPage, fetchAllMessages, fetchContact } from './conv-agent.js'; + +const $ = (id) => document.getElementById(id); +const params = new URLSearchParams(location.search); + +const state = { + tabId: Number(params.get('tabId')) || null, + origin: params.get('origin') || null, + locationId: null, + locationName: null, + dir: null, + writers: {}, + contacts: new Map(), + done: new Set(), + cursor: null, + total: null, + counts: { conversations: 0, messages: 0, bytes: 0 }, + options: { files: true, jsonl: true, pretty: false, contacts: true }, + errors: [], + running: false, + paused: false, + startedAt: 0, + startedFrom: 0 +}; + +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); + +function say(node, message, kind) { + node.textContent = message; + node.className = 'state' + (kind ? ' ' + kind : ''); +} + +function slug(name, fallback) { + const base = String(name || '') + .toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '').slice(0, 48); + return base || fallback; +} + +function openStep(n) { + $('s' + n).classList.remove('is-off'); +} + +/* ------------------------------------------------------------ page setup --- */ + +async function init() { + if (!state.tabId) { + $('sub').textContent = 'Open this page from the extension popup so it knows which tab to read.'; + return; + } + try { + const tab = await chrome.tabs.get(state.tabId); + // tab.url is only populated when the extension holds permission for it, so + // the popup passes the origin along rather than us asking for "tabs". + if (!state.origin && tab.url) state.origin = new URL(tab.url).origin; + } catch (e) { + $('sub').textContent = 'That HighLevel tab is gone. Reopen this page from the popup.'; + return; + } + if (!state.origin) { + $('sub').textContent = 'Missing origin. Reopen this page from the popup.'; + return; + } + $('sub').textContent = state.origin; + const granted = await chrome.permissions.contains({ origins: [state.origin + '/*'] }); + if (granted) await afterGrant(); +} + +async function grant() { + try { + const ok = await chrome.permissions.request({ origins: [state.origin + '/*'] }); + if (!ok) return say($('grantState'), 'Permission declined.', 'err'); + await afterGrant(); + } catch (err) { + say($('grantState'), String(err.message || err), 'err'); + } +} + +async function afterGrant() { + const probe = await inTab(probePage); + if (!probe || !probe.ok) { + return say($('grantState'), 'Could not read the sub-account from that tab. Reload it and try again.', 'err'); + } + state.locationId = probe.locationId; + state.locationName = probe.locationName; + $('sub').textContent = (probe.locationName || probe.locationId) + ' · ' + state.origin; + say($('grantState'), 'Access granted.', 'ok'); + $('s1').classList.add('is-done'); + openStep(2); +} + +async function inTab(func, args) { + const [result] = await chrome.scripting.executeScript({ + target: { tabId: state.tabId }, world: 'MAIN', func, args: args || [] + }); + return result ? result.result : undefined; +} + +/* --------------------------------------------------------------- folder --- */ + +async function pickFolder() { + try { + state.dir = await window.showDirectoryPicker({ mode: 'readwrite', id: 'ghl-conversation-archive' }); + } catch (e) { + return; // user cancelled + } + try { + const resumed = await loadManifest(); + say($('pickState'), + resumed + ? 'Resuming: ' + state.done.size + ' conversation(s) already archived here.' + : 'Folder ready.', + 'ok'); + $('s2').classList.add('is-done'); + openStep(3); + if (resumed) $('start').textContent = 'Resume'; + paint(); + } catch (err) { + say($('pickState'), String(err.message || err), 'err'); + } +} + +async function readJson(name) { + try { + const handle = await state.dir.getFileHandle(name); + const text = await (await handle.getFile()).text(); + return JSON.parse(text); + } catch (e) { + return null; + } +} + +async function loadManifest() { + const manifest = await readJson('manifest.json'); + if (!manifest) return false; + state.cursor = manifest.cursor || null; + state.total = manifest.total || null; + state.counts = manifest.counts || state.counts; + state.done = new Set(manifest.doneConversationIds || []); + state.errors = manifest.errors || []; + const contacts = await readJson('contacts.json'); + if (contacts) for (const [id, c] of Object.entries(contacts)) state.contacts.set(id, c); + return state.done.size > 0; +} + +/** + * Append-mode writer for the two JSONL streams. Opening with keepExistingData + * and seeking to the current end is what makes a resumed run add to the file + * instead of truncating everything the last run wrote. + */ +async function openWriter(name) { + const handle = await state.dir.getFileHandle(name, { create: true }); + const size = (await handle.getFile()).size; + const stream = await handle.createWritable({ keepExistingData: true }); + await stream.seek(size); + return stream; +} + +async function openWriters() { + if (state.options.jsonl) state.writers.messages = await openWriter('messages.jsonl'); + state.writers.conversations = await openWriter('conversations.jsonl'); +} + +async function closeWriters() { + for (const key of Object.keys(state.writers)) { + try { await state.writers[key].close(); } catch (e) { /* already closed */ } + delete state.writers[key]; + } +} + +async function writeJson(name, value) { + const handle = await state.dir.getFileHandle(name, { create: true }); + const stream = await handle.createWritable(); // truncates: these are rewritten whole + await stream.write(JSON.stringify(value, null, 2) + '\n'); + await stream.close(); +} + +async function checkpoint() { + await writeJson('manifest.json', { + locationId: state.locationId, + locationName: state.locationName, + options: state.options, + total: state.total, + counts: state.counts, + cursor: state.cursor, + complete: !state.running && state.total !== null && state.counts.conversations >= (state.total || 0), + errors: state.errors, + doneConversationIds: [...state.done] + }); + await writeJson('contacts.json', Object.fromEntries(state.contacts)); +} + +/* ------------------------------------------------------------------ run --- */ + +async function start() { + state.options = { + files: $('optFiles').checked, + jsonl: $('optJsonl').checked, + pretty: $('optPretty').checked, + contacts: $('withContacts').checked + }; + if (!state.options.files && !state.options.jsonl) { + return say($('runState'), 'Pick at least one output format.', 'err'); + } + state.running = true; + state.paused = false; + state.startedAt = Date.now(); + state.startedFrom = state.counts.conversations; + $('start').hidden = true; + $('pause').hidden = false; + $('stop').hidden = false; + $('stats').hidden = false; + $('barWrap').hidden = false; + say($('runState'), ''); + + try { + await openWriters(); + await crawl(); + } catch (err) { + say($('runState'), String(err.message || err), 'err'); + } finally { + await closeWriters(); + await checkpoint(); + state.running = false; + $('pause').hidden = true; + $('stop').hidden = true; + $('start').hidden = false; + $('start').textContent = 'Resume'; + paint(); + } +} + +async function crawl() { + let queue = []; + + while (state.running) { + if (state.paused) { await sleep(300); continue; } + + if (!queue.length) { + $('now').textContent = 'Fetching the next page of conversations…'; + const page = await inTab(fetchConversationsPage, [state.locationId, state.cursor, 100]); + if (!page || !page.ok) throw new Error('Conversation list failed: ' + ((page && page.error) || 'unknown')); + if (typeof page.total === 'number') state.total = page.total; + if (!page.conversations.length) { + say($('runState'), 'Done. Every conversation in this sub-account is archived.', 'ok'); + state.running = false; + break; + } + queue = page.conversations; + state.cursor = page.cursor; + } + + const conversation = queue.shift(); + const id = conversation.id; + if (state.done.has(id)) continue; + + $('now').textContent = 'Archiving ' + (conversation.contactName || conversation.fullName || id); + + const result = await inTab(fetchAllMessages, [id, 200]); + if (!result || !result.ok) { + state.errors.push({ id, name: conversation.contactName || null, error: (result && result.error) || 'unknown' }); + renderErrors(); + state.done.add(id); // recorded as failed; a rerun should not stall here forever + continue; + } + + if (state.options.contacts && conversation.contactId && !state.contacts.has(conversation.contactId)) { + const c = await inTab(fetchContact, [conversation.contactId]); + if (c && c.ok) state.contacts.set(conversation.contactId, c.contact); + await sleep(40); + } else if (conversation.contactId && !state.contacts.has(conversation.contactId)) { + state.contacts.set(conversation.contactId, { + id: conversation.contactId, + name: conversation.contactName || conversation.fullName || null, + email: conversation.email || null, + phone: null + }); + } + + await writeConversation(conversation, result); + + state.counts.conversations++; + state.counts.messages += result.messages.length; + state.done.add(id); + + if (state.counts.conversations % 10 === 0) await checkpoint(); + paint(); + await sleep(90); // deliberate throttle; a 20-request burst was fine but this runs for thousands + } +} + +async function writeConversation(conversation, result) { + const name = 'conversations/' + slug(conversation.contactName || conversation.fullName, 'contact') + + '-' + String(conversation.id).slice(0, 8) + '.json'; + let written = 0; + + if (state.options.files) { + const record = { + conversation, + messageCount: result.messages.length, + pages: result.pages, + truncated: result.truncated || false, + messages: result.messages + }; + const folder = await state.dir.getDirectoryHandle('conversations', { create: true }); + const handle = await folder.getFileHandle(name.split('/')[1], { create: true }); + const stream = await handle.createWritable(); + const text = state.options.pretty + ? JSON.stringify(record, null, 2) + '\n' + : JSON.stringify(record) + '\n'; + await stream.write(text); + await stream.close(); + written += text.length; + } + + const convLine = JSON.stringify({ + ...conversation, + messageCount: result.messages.length, + file: state.options.files ? name : null + }) + '\n'; + await state.writers.conversations.write(convLine); + written += convLine.length; + + if (state.options.jsonl) { + // Denormalised on purpose: each line carries the contact identity so the + // file stands alone in SQLite without a join back to conversations.jsonl. + for (const message of result.messages) { + const line = JSON.stringify({ + ...message, + conversationId: conversation.id, + contactId: conversation.contactId, + contactName: conversation.contactName || conversation.fullName || null + }) + '\n'; + written += line.length; + await state.writers.messages.write(line); + } + } + + state.counts.bytes += written; +} + +/* ------------------------------------------------------------------- ui --- */ + +function paint() { + $('nConv').textContent = state.counts.conversations.toLocaleString(); + $('nMsg').textContent = state.counts.messages.toLocaleString(); + $('nMb').textContent = (state.counts.bytes / 1048576).toFixed(1); + + if (state.total) { + const pct = Math.min(100, (state.counts.conversations / state.total) * 100); + $('bar').style.width = pct.toFixed(1) + '%'; + } + + if (state.total && state.counts.conversations > 3) { + const projected = (state.counts.bytes / state.counts.conversations) * state.total; + const gb = projected / 1073741824; + $('proj').textContent = gb >= 1 ? gb.toFixed(2) + ' GB' : Math.round(projected / 1048576) + ' MB'; + } + + const doneThisRun = state.counts.conversations - state.startedFrom; + if (state.running && doneThisRun > 5 && state.total) { + const perItem = (Date.now() - state.startedAt) / doneThisRun; + const left = Math.max(0, state.total - state.counts.conversations) * perItem; + $('eta').textContent = humanize(left); + } +} + +function humanize(ms) { + const mins = Math.round(ms / 60000); + if (mins < 1) return '<1 min'; + if (mins < 60) return mins + ' min'; + return Math.floor(mins / 60) + 'h ' + (mins % 60) + 'm'; +} + +function renderErrors() { + $('errBox').hidden = state.errors.length === 0; + $('errCount').textContent = String(state.errors.length); + $('errList').textContent = ''; + for (const e of state.errors.slice(-25)) { + const li = document.createElement('li'); + li.textContent = (e.name || e.id) + ' — ' + e.error; + $('errList').append(li); + } +} + +$('grant').addEventListener('click', grant); +$('pick').addEventListener('click', pickFolder); +$('start').addEventListener('click', start); +$('pause').addEventListener('click', () => { + state.paused = !state.paused; + $('pause').textContent = state.paused ? 'Continue' : 'Pause'; + say($('runState'), state.paused ? 'Paused. The folder is consistent; you can close this tab.' : ''); +}); +$('stop').addEventListener('click', () => { state.running = false; }); +window.addEventListener('beforeunload', (e) => { + if (state.running) { e.preventDefault(); e.returnValue = ''; } +}); + +init(); diff --git a/ghl-conversation-archive/conv-agent.js b/ghl-conversation-archive/conv-agent.js new file mode 100644 index 0000000..7763da1 --- /dev/null +++ b/ghl-conversation-archive/conv-agent.js @@ -0,0 +1,174 @@ +// Injected into the HighLevel page's MAIN world, same contract as agent.js: +// each function is serialized and re-parsed there, so nothing may reference +// module scope. The helper is repeated in each function for that reason. +// +// Conversation endpoints, all verified against a live sub-account: +// GET {svc}/conversations/search?locationId=&limit=&startAfterDate= +// -> { conversations: [...], total } +// Paging is a cursor, NOT an offset. `offset` and `page` are accepted +// and silently ignored; the cursor is the previous page's last +// conversation's `sort[0]` value, passed as startAfterDate. +// GET {svc}/conversations/{id}/messages?limit=&lastMessageId= +// -> { messages: { messages: [...], lastMessageId, nextPage } } +// GET {svc}/contacts/{id} -> full contact record +// +// There is also GET {svc}/conversations/messages/export, which is the +// purpose-built bulk endpoint, but it rejects a browser session with +// "Can not fetch messages from non-OAuth channel". It needs a Private +// Integration or OAuth token, so this module does not use it. + +const SVC = 'https://services.leadconnectorhq.com'; + +/** One page of the conversation list, plus the cursor for the next page. */ +export async function fetchConversationsPage(locationId, startAfterDate, limit) { + const request = async (url, params) => { + const store = window.SHELL_STORE; + if (store && store.$http) return (await store.$http.get(url, { params })).data; + const findJwt = (node, depth) => { + if (!node || depth > 4) return null; + if (typeof node === 'string') { + return /^ey[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\./.test(node) ? node : null; + } + if (typeof node !== 'object') return null; + for (const key of Object.keys(node)) { + const hit = findJwt(node[key], depth + 1); + if (hit) return hit; + } + return null; + }; + const token = findJwt(store && store.state && store.state.auth, 0); + const target = new URL(url); + for (const [k, v] of Object.entries(params || {})) { + if (v !== undefined && v !== null) target.searchParams.set(k, String(v)); + } + const res = await fetch(target.toString(), { + headers: { Authorization: 'Bearer ' + token, channel: 'APP', source: 'WEB_USER', Version: '2021-04-15' } + }); + if (!res.ok) throw new Error('HTTP ' + res.status); + return res.json(); + }; + + try { + const params = { locationId, limit: limit || 100 }; + if (startAfterDate) params.startAfterDate = startAfterDate; + const data = await request('https://services.leadconnectorhq.com/conversations/search', params); + const list = (data && data.conversations) || []; + const last = list[list.length - 1]; + return { + ok: true, + total: data && data.total, + conversations: list, + cursor: last && Array.isArray(last.sort) ? last.sort[0] : null + }; + } catch (err) { + const status = err && err.response && err.response.status; + return { ok: false, error: status ? 'HTTP ' + status : String((err && err.message) || err) }; + } +} + +/** + * Every message in one conversation, walking the lastMessageId cursor. + * Looping inside the page rather than round-tripping per page matters: a + * 580-message thread is 6 pages, and each executeScript hop costs more than + * the request it carries. + */ +export async function fetchAllMessages(conversationId, maxPages) { + const request = async (url, params) => { + const store = window.SHELL_STORE; + if (store && store.$http) return (await store.$http.get(url, { params })).data; + const findJwt = (node, depth) => { + if (!node || depth > 4) return null; + if (typeof node === 'string') { + return /^ey[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\./.test(node) ? node : null; + } + if (typeof node !== 'object') return null; + for (const key of Object.keys(node)) { + const hit = findJwt(node[key], depth + 1); + if (hit) return hit; + } + return null; + }; + const token = findJwt(store && store.state && store.state.auth, 0); + const target = new URL(url); + for (const [k, v] of Object.entries(params || {})) { + if (v !== undefined && v !== null) target.searchParams.set(k, String(v)); + } + const res = await fetch(target.toString(), { + headers: { Authorization: 'Bearer ' + token, channel: 'APP', source: 'WEB_USER', Version: '2021-04-15' } + }); + if (!res.ok) throw new Error('HTTP ' + res.status); + return res.json(); + }; + + const cap = maxPages || 200; + const url = 'https://services.leadconnectorhq.com/conversations/' + conversationId + '/messages'; + const all = []; + let cursor = null; + let pages = 0; + let truncated = false; + + try { + for (; pages < cap; pages++) { + const params = { limit: 100 }; + if (cursor) params.lastMessageId = cursor; + const data = await request(url, params); + const block = (data && data.messages) || {}; + const batch = block.messages || []; + all.push(...batch); + cursor = block.lastMessageId; + if (!block.nextPage || !batch.length || !cursor) { pages++; break; } + await new Promise((r) => setTimeout(r, 60)); + } + if (pages >= cap) truncated = true; + return { ok: true, messages: all, pages, truncated }; + } catch (err) { + const status = err && err.response && err.response.status; + return { ok: false, error: status ? 'HTTP ' + status : String((err && err.message) || err), messages: all, pages }; + } +} + +/** Full contact record. Only the identity fields a later migration would map on. */ +export async function fetchContact(contactId) { + const request = async (url) => { + const store = window.SHELL_STORE; + if (store && store.$http) return (await store.$http.get(url, { headers: { Version: '2021-07-28' } })).data; + const findJwt = (node, depth) => { + if (!node || depth > 4) return null; + if (typeof node === 'string') { + return /^ey[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\./.test(node) ? node : null; + } + if (typeof node !== 'object') return null; + for (const key of Object.keys(node)) { + const hit = findJwt(node[key], depth + 1); + if (hit) return hit; + } + return null; + }; + const token = findJwt(store && store.state && store.state.auth, 0); + const res = await fetch(url, { + headers: { Authorization: 'Bearer ' + token, channel: 'APP', source: 'WEB_USER', Version: '2021-07-28' } + }); + if (!res.ok) throw new Error('HTTP ' + res.status); + return res.json(); + }; + + try { + const data = await request('https://services.leadconnectorhq.com/contacts/' + contactId); + const c = (data && data.contact) || data || {}; + return { + ok: true, + contact: { + id: c.id || contactId, + firstName: c.firstName || null, + lastName: c.lastName || null, + email: c.email || null, + phone: c.phone || null, + dateAdded: c.dateAdded || null, + tags: c.tags || [] + } + }; + } catch (err) { + const status = err && err.response && err.response.status; + return { ok: false, error: status ? 'HTTP ' + status : String((err && err.message) || err) }; + } +} diff --git a/ghl-conversation-archive/icons/icon128.png b/ghl-conversation-archive/icons/icon128.png new file mode 100644 index 0000000..855bc32 Binary files /dev/null and b/ghl-conversation-archive/icons/icon128.png differ diff --git a/ghl-conversation-archive/icons/icon16.png b/ghl-conversation-archive/icons/icon16.png new file mode 100644 index 0000000..4f4f82a Binary files /dev/null and b/ghl-conversation-archive/icons/icon16.png differ diff --git a/ghl-conversation-archive/icons/icon48.png b/ghl-conversation-archive/icons/icon48.png new file mode 100644 index 0000000..07441e9 Binary files /dev/null and b/ghl-conversation-archive/icons/icon48.png differ diff --git a/ghl-conversation-archive/manifest.json b/ghl-conversation-archive/manifest.json new file mode 100644 index 0000000..81530b3 --- /dev/null +++ b/ghl-conversation-archive/manifest.json @@ -0,0 +1,27 @@ +{ + "manifest_version": 3, + "name": "GHL Conversation Archive", + "version": "1.0.0", + "description": "Archives every conversation and message in a HighLevel sub-account to a folder on your machine. Export only.", + "permissions": [ + "activeTab", + "scripting" + ], + "optional_host_permissions": [ + "https://*/*" + ], + "action": { + "default_title": "Archive conversations", + "default_popup": "popup.html", + "default_icon": { + "16": "icons/icon16.png", + "48": "icons/icon48.png", + "128": "icons/icon128.png" + } + }, + "icons": { + "16": "icons/icon16.png", + "48": "icons/icon48.png", + "128": "icons/icon128.png" + } +} diff --git a/ghl-conversation-archive/popup.css b/ghl-conversation-archive/popup.css new file mode 100644 index 0000000..3c94722 --- /dev/null +++ b/ghl-conversation-archive/popup.css @@ -0,0 +1,28 @@ +:root { + color-scheme: light dark; + --bg: #ffffff; --fg: #16181d; --muted: #6b7280; --line: #e4e7ec; + --accent: #107c66; --accent-fg: #ffffff; --err: #b42318; +} +@media (prefers-color-scheme: dark) { + :root { + --bg: #16181d; --fg: #f2f4f7; --muted: #98a2b3; --line: #2c3038; + --accent: #4fc7a8; --accent-fg: #0b0d11; --err: #ff9a92; + } +} +* { box-sizing: border-box; } +body { + width: 300px; margin: 0; padding: 16px; background: var(--bg); color: var(--fg); + font: 13px/1.5 system-ui, -apple-system, "Segoe UI", sans-serif; +} +h1 { margin: 0; font-size: 15px; font-weight: 600; } +p { margin: 4px 0 0; } +.muted { color: var(--muted); font-size: 12px; } +header { padding-bottom: 10px; border-bottom: 1px solid var(--line); margin-bottom: 10px; } +button.primary { + width: 100%; margin-top: 10px; padding: 9px 12px; border: 0; border-radius: 7px; + background: var(--accent); color: var(--accent-fg); font: inherit; font-weight: 600; cursor: pointer; +} +button.primary:disabled { opacity: .45; cursor: default; } +#status { font-size: 12px; } +#status.err { color: var(--err); } +.foot { margin-top: 12px; padding-top: 10px; border-top: 1px solid var(--line); } diff --git a/ghl-conversation-archive/popup.html b/ghl-conversation-archive/popup.html new file mode 100644 index 0000000..67e9ef2 --- /dev/null +++ b/ghl-conversation-archive/popup.html @@ -0,0 +1,19 @@ + + + + +Conversation Archive + + + +
+

Conversation Archive

+

Checking this tab…

+
+

+ +

+

Export only. Nothing in this sub-account is modified.

+ + + diff --git a/ghl-conversation-archive/popup.js b/ghl-conversation-archive/popup.js new file mode 100644 index 0000000..832a8e3 --- /dev/null +++ b/ghl-conversation-archive/popup.js @@ -0,0 +1,57 @@ +import { probePage } from './agent.js'; +import { fetchConversationsPage } from './conv-agent.js'; + +const $ = (id) => document.getElementById(id); +let context = null; + +const REASONS = { + 'not-highlevel': 'Open a HighLevel sub-account tab, then reopen this popup.', + 'no-location': 'No sub-account detected. Navigate into a location first.', + 'no-auth': 'Could not reach the app session. Reload the page and try again.' +}; + +async function activeTab() { + const [tab] = await chrome.tabs.query({ active: true, currentWindow: true }); + return tab; +} + +async function inMainWorld(tabId, func, args) { + const [result] = await chrome.scripting.executeScript({ + target: { tabId }, world: 'MAIN', func, args: args || [] + }); + return result ? result.result : undefined; +} + +async function init() { + try { + const tab = await activeTab(); + const probe = await inMainWorld(tab.id, probePage); + if (!probe || !probe.ok) { + $('target').textContent = REASONS[probe && probe.reason] || 'This tab is not a HighLevel app page.'; + return; + } + context = { tabId: tab.id, origin: new URL(tab.url).origin, ...probe }; + $('target').textContent = probe.locationName || ('Sub-account ' + probe.locationId); + $('open').disabled = false; + + // One cheap call, purely so the size is not a surprise mid-run. + const page = await inMainWorld(tab.id, fetchConversationsPage, [probe.locationId, null, 1]); + $('scale').textContent = page && page.ok && typeof page.total === 'number' + ? page.total.toLocaleString() + ' conversations to archive.' + : ''; + } catch (err) { + $('target').textContent = 'Cannot read this tab.'; + $('status').textContent = String(err.message || err); + $('status').className = 'err'; + } +} + +$('open').addEventListener('click', async () => { + const url = chrome.runtime.getURL('archive.html') + + '?tabId=' + encodeURIComponent(context.tabId) + + '&origin=' + encodeURIComponent(context.origin); + await chrome.tabs.create({ url }); + window.close(); +}); + +init();