v0.3.1: OpenCLI Browser Bridge + x-article + substack connectors#3
Merged
Conversation
## What Real-browser draft creation for providers that have no public API (starting with X Articles; Substack lands in PR-B). Built on Playwright as an opt-in extra. ## Components ### `core/browser.py` — session manager (foundation) - `BrowserNotInstalledError` / `BrowserStateMissingError` (both `MMPError`) - `state_path(provider)` / `state_exists(provider)` / `delete_state(provider)` - `browser_context(provider, headless=True, require_state=True)` — context manager yielding a logged-in `BrowserContext` - `login_interactive(provider, login_url)` — headed flow, captures state and persists to `~/.config/mmp/browser-state/<provider>.json` (chmod 600) - Lazy import: missing playwright surfaces as `BrowserNotInstalledError` with install instructions, not at module import time ### `Provider.browser_login_url` (optional class attr) Providers that authenticate via browser session set this. Used by `mmp browser login <name>` to know where to navigate. ### `mmp browser` CLI subcommand ```bash mmp browser login <provider> # headed flow, capture session mmp browser status [provider] # list saved states with mtime mmp browser logout <provider> # delete state ``` ### `providers/x_article/` upgraded - `browser_login_url = "https://x.com/i/flow/login"` - `internal/browser_flow.py` — actual Playwright draft flow (selectors isolated as module-top constants for easy patching when X drifts UI) - `provider.execute(mode="draft")`: - if no state OR no playwright → `mode_actual="stub"`, writes `TODO-connector.md` with both setup and manual fallback paths - if both available → `mode_actual="draft-platform"` with real X article ID as `external_id` ### `docs/browser-connectors.md` Setup, session lifecycle, troubleshooting, security guidance, selector drift remediation, roadmap. ## Tests - `tests/core/test_browser.py` — 9 unit tests for browser module (state paths, missing-state errors, install hint, chmod 600 on save) - `tests/integration/test_browser_cli.py` — 7 tests for `mmp browser` subcommand (status, logout, login validation) - `providers/x_article/tests/test_provider.py` — 4 new tests (stub fallback when no state; mocked browser flow returns draft-platform; health_check reflects state; browser_login_url is set) All tests pass without playwright installed (the foundation is testable with mocks). Real-browser tests are gated on the `[browser]` extra + real X account; documented in `docs/manual-verification.md` (next PR). ## Out of this PR - Substack connector (PR-B; same pattern, different selectors) - Real-account verification of x-article (depends on user with X account capturing login state) - Selector drift detection / auto re-login prompt (v0.4) ## Migration Zero-impact for v0.3.0 users. The `[browser]` extra is opt-in. Without it, x-article keeps returning `mode_actual="stub"` exactly as before v0.3.1, just with a more helpful TODO-connector.md.
Real-account verification of x-article hit two showstoppers with the Playwright approach: 1. Google OAuth (sign-in path on X) detected Playwright Chromium and refused login. The user couldn't sign in via the fresh Chromium. 2. CDP-attach to the user's real Chrome required re-launching Chrome with `--remote-debugging-port` AND a non-default `--user-data-dir` (Chrome refuses CDP on the default profile). Heavy friction. Pivot to OpenCLI (https://github.com/jackwener/opencli): a Chrome extension + Node.js daemon that exposes browser primitives via `opencli browser ...`. The extension drives the user's real Chrome — no automation detection trips, no separate session, no Chrome relaunch. ## Architecture - `core/browser.py` — wraps `opencli browser` subcommands as a Python module. Functions: `open_url`, `state`, `get_url`, `get_title`, `click`, `type_text`, `wait`, `evaluate`, `screenshot`, `find`, `is_connected`, `doctor`. Lazy `npx -y @jackwener/opencli` invocation if `opencli` not on PATH. - `core/provider.Provider.browser_login_url` — informational anchor for `mmp browser login <provider>`; opens that URL in Chrome (the user logs in normally, mmp doesn't manage state). - `mmp browser` CLI subcommand: - `mmp browser status` — extension connectivity check - `mmp browser login <provider>` — opens provider's login URL - `mmp browser doctor` — full opencli diagnostic - `providers/x_article/internal/browser_flow.py` — drives X Articles compose flow: 1. Navigate to `x.com/compose/articles` (drafts list) 2. Click "Write" / "撰写" → X allocates draft, URL becomes `/compose/articles/edit/<draft-id>` 3. Capture draft_id from URL (= our `external_id`) 4. Type title into `textarea[placeholder="添加标题"]` (with English fallback `Add a title`) 5. Type body into `[data-testid="composer"]` 6. X auto-saves; we sleep 4s to let it land - `providers/x_article/provider.execute(draft)` — calls `is_connected()`, falls back to "stub" mode (writes TODO-connector.md) if bridge not connected. Real flow returns `mode_actual="draft-platform"`. ## Real-account verification Verified end-to-end against Lewis's X Premium account: ``` mode_actual: draft-platform external_id: 2051993962379132929 draft_url: https://x.com/compose/articles/edit/2051993962379132929 ``` Draft confirmed in X drafts list. ## Bugs found & fixed during verification 1. v0.2 Playwright path tried `[data-testid="articleTitle"]` — X has no such element. The articles compose UI is at `/compose/articles` not `/i/articles/compose` (the latter returns 404 even for Premium). 2. `[data-testid="tweetTextarea_0"]` matched 2 elements (modal + inline timeline composer). Scoped with `div[role="dialog"] ` prefix. 3. `browser state` emits text not JSON; URL field was always empty. Added `get_url()` / `get_title()` helpers using `browser get url` which returns plain text. 4. X Articles compose has no "Save Draft" button — autosave fires while typing. Removed unnecessary save-button click. ## Removed - `pyproject.toml [project.optional-dependencies] browser = [playwright]` is gone. No Python deps for browser flow now (opencli is Node.js). - `playwright` mypy override (no longer needed). - All Playwright-specific code in `core/browser.py` (login_interactive with stdin/auto-detect/CDP modes, BrowserStateMissingError, etc). ## Documentation `docs/browser-connectors.md` rewritten end-to-end: - OpenCLI install + extension setup - Setup verification flow - Selector-drift remediation playbook - i18n caveats (selectors target Chinese UI; English fallback included) - Security model (cookies stay in user's Chrome) - Roadmap (v0.3.2 substack, v0.4 session expiry detection)
Same OpenCLI Browser Bridge backend as x-article, applied to Substack.
First-shot verification (no selector debugging needed) — the abstraction
from PR-A carried over cleanly.
## Architecture
- `providers/substack/internal/browser_flow.py` — drives Substack's
post composer:
1. Navigate to `<publication>.substack.com/publish/post`
2. Substack auto-allocates draft, URL becomes
`/publish/post/<draft-id>` (numeric)
3. Capture draft_id from URL → `external_id`
4. Type title into `[data-testid="post-title"]`
5. Type subtitle (optional) into `input[placeholder="Add a subtitle…"]`
with English fallback `Add a subtitle...` (no Unicode ellipsis)
6. Type body into `[data-testid="editor"]` (contenteditable)
7. Substack autosave; sleep 4s
- `providers/substack/provider.py` — orchestration mirrors x-article
with one Substack-specific addition: **publication URL discovery**.
Each Substack writer has a unique subdomain (e.g.
`lewisxbt.substack.com`); we resolve it via:
1. Manifest target's `options.publication_url`
2. `SUBSTACK_PUBLICATION_URL` env var
3. Stub fallback with clear remediation message
No universal `substack.com/publish/post` URL works for arbitrary
users (it redirects to "Discover Substack Newsletters").
## Real-account verification ✅
Tested end-to-end against Lewis's Substack publication
`lewisxbt.substack.com`:
```
mode_actual: draft-platform
external_id: 196650004
draft_url: https://lewisxbt.substack.com/publish/post/196650004
```
Draft confirmed in Substack publish dashboard.
## Three-platform demo run ✅
Single `mmp publish examples/longform-multi.yaml --mode-override draft`
hit all three providers, all returned status=ok with real external IDs:
```
wechat-article: WbX2nHWvJ4Nnr7szvvzYx_6hRvH0gF526sLiaJJIHHXM-UPSfevgQ3GpIxsPvpDY (WeChat OA API)
x-article: 2051997319453929472 (X.com via OpenCLI)
substack: 196650004 (Substack via OpenCLI)
```
This is multi-media-publisher's first complete cross-platform draft run
end-to-end against real accounts.
## Tests
`providers/substack/tests/test_provider.py`: 9 tests
- `test_validate_passes`
- `test_validate_subtitle_too_long`
- `test_prepare_writes_payload`
- `test_execute_draft_stub_when_no_publication_url`
- `test_execute_draft_stub_when_bridge_disconnected`
- `test_execute_draft_uses_publication_url_from_target_options`
(manifest options take priority over env var)
- `test_execute_draft_invokes_browser_flow`
- `test_health_check_reflects_bridge`
- `test_browser_login_url_is_set`
- `test_execute_publish_refused`
## Backward compat
Manifests using substack as a target need a `publication_url` now
(via target options or env). Without it, fall back to stub mode with
a clear error. Existing v0.2/v0.3.0 manifests that didn't have
publication_url set continue to function (just stay in stub mode
exactly as they did before).
Nowhitestar
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
P1b complete: real-browser draft creation for both X Articles and Substack via the user's real Chrome (driven by OpenCLI Browser Bridge). Both connectors verified end-to-end against real accounts, no automation detection trips, no separate logins.
Originally built on Playwright + fresh Chromium. Pivoted to OpenCLI mid-PR after Google OAuth detection blocked verification — see commit
6ad2010for the rationale and migration.Three-platform demo ✅
Single
mmp publish examples/longform-multi.yaml --mode-override drafthit all three providers, all returned status=ok with real external IDs:mmp's actual user value end-to-end: one publish, three platforms, all draft-platform with real IDs. Verified against real WeChat OA + X Premium + Substack publication accounts.
What's in this PR
Foundation (
core/browser.py)opencli browsersubcommands as Python primitives:open_url,state,get_url,get_title,click,type_text,wait,evaluate,screenshot,find,is_connected,doctornpx -y @jackwener/opencliinvocation if the user doesn't have a global installBrowserNotInstalledError,BrowserNotConnectedError,BrowserCommandErrorProvider abstraction (
core/provider.Provider.browser_login_url)mmp browser login <provider>opens that URL in the user's ChromeCLI (
mmp browser)mmp browser status— extension connectivity checkmmp browser login <provider>— opens provider's login URLmmp browser doctor— full opencli diagnosticx-article connector
providers/x_article/internal/browser_flow.py— drives X Articles real flow2051993962379132929substack connector
providers/substack/internal/browser_flow.py— drives Substack's post composer196650004onlewisxbt.substack.comDocs
docs/browser-connectors.mdrewritten — OpenCLI setup, troubleshooting, selector-drift playbook, i18n caveats, security modelBugs found & fixed during real-account verification
/i/articles/composereturns 404 (even for X Premium)/compose/articles[data-testid="articleTitle"]doesn't exist in current X UItextarea[placeholder="添加标题"][data-testid="tweetTextarea_0"]matches 2 elements (modal + inline)div[role="dialog"]prefixbrowser statereturns text not JSON; URL field emptyget_url()/get_title()helpers/publish/postredirects to "Discover Newsletters" without publication URLTest Plan
make test— lint + format + mypy + 100+ tests + smoke2051993962379132929)196650004)Backward compat
Zero impact for v0.3.0 users:
publication_url— but those without it just stay in stub mode (same as v0.3.0)Out of this PR
x-postif interest🤖 Generated with Claude Code