feat: add structured remote control commands

[BrianMcBrayer]

Before Submitting This PR

• I have searched existing issues and pull requests (including closed ones) to ensure this isn't a duplicate
• I have read CONTRIBUTING.md

If this is a feature or change that was previously closed/rejected:

• I have explained in the description below why this should be reconsidered
• I have gathered community feedback (link to discussion below)

Human Written Description

This is my first time collaborating, but I wanted to say thanks for Handy. I use it on Windows every day. I have an elgator streamdeck, and I was thinking how useful it would be to use it with Handy -- one button toggles recording, another button copies a transcript, another button pastes, etc.

Related Issues/Discussions

Discussion: #211 (comment)

Related PRs: #792 established Handy's existing single-instance CLI controls. #659 addressed a similar latest-transcript retrieval use case through a new global shortcut; this proposal instead keeps bindings outside Handy and adds no shortcut or settings UI.

Community Feedback

This draft is being shared alongside a proposal in Discussion #211 to gather feedback before requesting review. I am happy to narrow or revise the command surface based on maintainer and community direction.

What Changed

• Add structured, typed commands for idempotent recording start/stop/toggle/cancel actions.
• Add commands to copy or paste the most recent completed transcript, preferring post-processed text when available.
• Allow a one-shot override using Handy's existing paste-method names without changing saved settings.
• Preserve the existing --toggle-transcription, --toggle-post-process, and --cancel flags as aliases.
• Reuse the existing single-instance transport, transcription coordinator, history manager, and paste pipeline; no new shortcut, UI, settings, server, or network surface is introduced.

Testing

All automated development and verification ran in Docker using Ubuntu 24.04:

• cargo test: 171 passed
• cargo fmt -- --check
• cargo clippy --all-targets (only pre-existing warnings remain)
• bun run check:translations
• bun run lint
• bunx prettier --check .
• bun run build
• Inspected generated CLI help and nested command parsing

Windows runtime and Stream Deck focus/paste behavior have not yet been validated. I would appreciate help using the draft PR's Windows build artifact for that target-platform testing before this is considered ready for review.

Screenshots/Videos (if applicable)

N/A — this is a CLI-only change with no new UI.

AI Assistance

• No AI was used in this PR
• AI was used (please describe below)

If AI was used:

• Tools used: OpenAI Codex
• How extensively: Codex helped research the existing architecture and maintainer context, design and implement the change, refine the code and tests, run Docker-isolated verification, and draft the non-human-written sections of this description. I directed the design and reviewed the command interface and public proposal. And I should add that I hand-reviewed all the code, did a few rounds with it, and also worked through all the messaging. I use AI a ton, but I own all my stuff and try not to lob vibe-coded slop. Hopefully I did that right here and please be as critical as needed.

[cjpais]

I just want to say I appreciate you submitting, but right now anything related to shortcuts are not going to be merged largely because this is effectively a UI UX challenge because 100 different people want 100 different features and shortcuts, so we need to figure out a generic solution, not a per-user solution

I'm closed because I don't understand your human written description in total. From what it reads like, it seems like it's related to assigning different actions different shortcuts, and right now this is not going to be merged because we are not accepting new features, especially in regards to shortcuts. If you can describe in better detail what exactly this is doing, it would help me review from a human perspective. I don't exactly understand the context that this is coming from. I do understand it's a CLI only change, but I need more context

[BrianMcBrayer]

@cjpais sounds good! I'll explain, but I know you're busy.

This is my take at generalizing the CLI so that it can eventually drive all the functionality that Handy can do if a person wanted that. My goal was to allow anyone using it to be able to drive it via CLI.

It would look something like the following, and I can refactor it if we even want it but just want different words:

handy recording start [--post-process]
handy recording stop
handy recording toggle [--post-process]

handy transcript clipboard
handy transcript paste

That's what this PR suggests. But it also doesn't back Handy into a corner and would allow the following in the future:

handy model select <model-id>
handy model download <model-id>

<etc.>

I wrote more about the idea here and submitted this PR as an example of what the implementation could look like: #211 (comment)

Also, if I can help with Handy in general in way that would save you time, let me know.

[cjpais]

re-opening for now, I will consider more

the only thing is, I do just wonder how much this should be it's own cli rather than handy specifically? handy is not really meant to be a cli

[BrianMcBrayer]

Thanks for reopening. And also, one of the reasons I like Handy is because you are thoughtful about the architecture, and I don't want to mess that up.

My engineering brain kept running with this in hopes I can help in some way, so I looked at the applications I use/have used and from what I can tell these are four different ways of doing UI + CLI

• CopyQ (tray app, same binary) and Superwhisper (separate CLI install) both end up needing some out-of-band control so Stream Deck / scripts can drive them without stuffing every binding into the app UI.
• Espanso is CLI/daemon-first by design
• OBS hosts a protocol and Stream Deck / obs-cli sit outside.
• 1Password ships a cli op as its own product separate from 1Password

The cheapest pattern I can see that doesn't back applications into a corner seems to be:

• one command surface
• execute inside the already-running app (mic, paste, history already live there)
• thin transport per OS.

This is similar to how Handy works today with its existing flags like --toggle-transcription, but it generalizes into NOUN VERB (like model download <model-id>) so the whole app can be CLI'd eventually.

The only way I can work out a separate CLI in my brain, Handy app would still need a handler and packaging/versioning of some sort, unless the CLI is separate product entirely that doesn't need Handy app installed.

^ those are just my musings from my research. I respect your application and just want to help, and if you like this at all I'm happy to change / extend / harden this PR.

[cjpais]

Yeah, I mean I guess more or less just to be clear is handy is really intended to be a desktop application and the CLI that we do offer is largely as a workaround for things on Linux rather than something explicitly supported. And I'm not sure I want to explicitly support a CLI or not because it just adds a lot of surface area to the app and features. So that's the main concern is the CLI that's there is really for extraordinarily experienced users, but really not the average person. And this app, broadly speaking, is targeted at the average person

[BrianMcBrayer]

Makes sense to me. I recommend it to my coworkers who don't want to have to work to get ASR going for that reason, so I'm in agreement there.

[cjpais]

Okay, so I want to thank you again for this and I just wanted to update you directly in this thread, which is I am now seriously considering adding the CLI. And in addition, broadly speaking, IPC to handy as well. So other applications potentially can hook into it in a reasonable way. I'm going to be thinking about this, and I'm just curious if you have any thoughts as well. I will probably start taking a deeper look at the PR myself in some time. I don't know if this PR itself will be merged, but certainly if I go implement IPC a CLI will come into focus and I will definitely be adding you as co-author as well

And in addition, if this is something that sounds interesting to you to potentially take on in the more full capacity, I'm happy to discuss with you or anyone else who wants to take part in the discussion of getting this implemented and brought into handy because I do think it is potentially useful to a lot of people or opens up possibilities that weren't there before and fundamentally does keep the application fairly simple. And in some ways, I know people have been asking for an API server, but I think IPC potentially solves this better. And if someone really wants to make an HTTP server on top of handy, they can fairly easily do that by using the IPC service. So I think this may be more broadly speaking can have a fair amount of impact, relatively speaking, and can kind of bring handy to a bunch of other applications for free

[BrianMcBrayer]

Thank you! I would definitely be interested in helping! I'll put some thought towards it and offer them; I'm also happy to just help with whatever architectural direction you come up with. Maintaining OSS is hard so I just want to help if I can.

IPC is a great idea, and I think it will be a lot easier to maintain / integrate upon than an API server. I'll come up with a couple of reference architectures to seed the discussion (I've found that anything, especially something with flaws, gets the conversation going better than nothing).

[BrianMcBrayer]

Today I thought through a few ways it could work. Feel free to shred it and I can continue to revise if it helps.

(note: AI helped me think it through but I discussed / vetted / hashed out / directed. We might decide it's slop but at least it's not vibe slop)

Short proposal

I suggest a small local connection that other programs on the same computer can use to talk to the running Handy application.

• Use a Unix socket on macOS/Linux and a named pipe on Windows.
• Restrict the connection to the current operating-system user.
• Start it automatically while Handy is running. Do not open a network port.
• Use one small, versioned JSON-RPC protocol. JSON-RPC is a common format for sending a named request and getting a reply. Each message would be one line of JSON.
• Keep Handy's existing recording pipeline in control. IPC should use the same internal actions as Handy itself and report what actually happened.

The first version would let a local program:

• Find which IPC features the running Handy version supports.
• Ask whether Handy is idle, recording, or processing.
• Start, stop, or cancel recording.
• Receive updates when Handy's state changes, a transcript finishes, or something fails.
• Receive a transcript ID and use it to fetch the finished text.

This would be enough for another local program to show Handy's real status, control recording, and use a completed transcript without pretending to type or reading Handy's database directly.

The same foundation could allow more in the future: live partial text, model or profile control, broader history access, audio access, or a separate HTTP bridge. This proposal does not include any of those things yet. Each could be considered later on its own.

Main options

Option	Pros	Cons
Unix sockets on macOS/Linux and named pipes on Windows	No network port; access can be limited to the current user; supports commands, replies, and updates; one shared protocol on every OS	Handy still needs connection code for both kinds of operating system and a public protocol to maintain
Native IPC for each OS, such as D-Bus and XPC	Best fit with each operating system	Three different public interfaces, client implementations, and sets of documentation
Localhost HTTP and WebSocket	Very easy for other applications and languages to use	Adds a server, port discovery, authentication, and browser/network security concerns to Handy

Would you be willing to review this direction and tell me what you would simplify or change? I am happy to follow your architectural direction and help with a small proof or the implementation if that would be useful.

---

Detailed proposal

Goals

• Let local applications control Handy and receive transcription results.
• Keep Handy as a normal desktop application, not a network service.
• Use the same public behavior on Windows, macOS, and Linux.
• Keep the first version small enough to maintain.
• Route all requests through Handy's existing managers and recording coordinator.

What version 1 would do

• Let a local program connect while Handy is running.
• Tell that program which IPC features this Handy version supports.
• Report Handy's current recording state.
• Accept start, stop, and cancel requests.
• Send updates when the state changes or work finishes or fails.
• Let the program fetch a completed transcript by its ID.

What this could allow later, but does not do yet

Once there is a stable local connection, separate future proposals could add:

• Live partial transcripts while the user speaks.
• Choosing models, languages, prompts, or saved profiles.
• Searching or editing more of transcript history.
• Reading recorded audio.
• More advanced workflow or accessibility integrations.
• A separate program that provides HTTP or another network API on top of Handy's local IPC.

None of those are included in version 1. This RFC only tries to leave a clean path for them if they are wanted later.

Version 1 specifically does not include:

• Changing persistent settings
• Selecting or downloading models
• Searching or editing all transcript history
• Reading audio files
• Live partial transcripts
• Loading third-party code inside Handy
• Listening on TCP or exposing an HTTP server

Architecture

flowchart LR
    Client[Local program] <-->|JSON-RPC messages| IPC[IPC connection layer]
    IPC --> Commands[Shared Handy actions]
    Commands --> Coordinator[TranscriptionCoordinator]
    Commands --> History[HistoryManager]
    Coordinator --> Pipeline[Audio and transcription pipeline]
    Coordinator --> Events[Application event broadcaster]
    History --> Events
    Events --> IPC

Loading

The IPC connection layer should only handle connections, JSON messages, and the updates each client wants. It should not contain separate recording or history logic.

The shared Handy actions should be used by IPC and Handy's own user interface and input paths. Updates should come from real application state changes, not from guesses made by the IPC connection layer.

Transport

Platform	Transport
macOS	User-private Unix-domain socket
Linux	User-private Unix-domain socket, preferably under XDG_RUNTIME_DIR
Windows	Named pipe restricted to the current user

The connection name should be stable and documented so another program can find it without a separate discovery file.

On startup, Handy should:

1. Initialize the coordinator and managers.
2. Open the local connection with current-user-only access.
3. Start accepting connections.
4. Continue normal desktop operation if IPC cannot start, while logging a clear error.

After a crash, the Unix implementation should test an existing socket before removing it. It should only remove the socket when no live Handy process answers. Windows removes a named pipe when its owner closes the last handle.

Protocol

Use JSON-RPC 2.0 over a full-duplex local connection, meaning the same connection carries messages in both directions. Each line contains one complete JSON object. Newlines inside transcript text are escaped by JSON.

Initial protocol rules:

• The first request must be handy.initialize.
• Each request has a unique id within that connection.
• Responses use the same id.
• Events are JSON-RPC notifications and do not have an id.
• JSON-RPC batch requests are not supported in version 1.
• Reject messages larger than 1 MiB before parsing them.
• Limit how many waiting updates Handy keeps for each client. Disconnect a client that cannot keep up.
• Never write transcript text to IPC diagnostic logs.

Connection setup

The client identifies itself, offers the protocol versions it supports, and asks for event topics.

{"jsonrpc":"2.0","id":1,"method":"handy.initialize","params":{"protocol_versions":[1],"client":{"name":"example-integration","version":"0.1.0"},"subscriptions":["state","transcription","errors"]}}

Handy selects a protocol version and returns its current real state and supported features.

{"jsonrpc":"2.0","id":1,"result":{"protocol_version":1,"server":{"name":"Handy","version":"0.9.5"},"session_id":"01K2ABC","capabilities":{"recording_control":true,"state_events":true,"transcript_fetch":true,"partial_transcripts":false,"configuration":false},"state":{"revision":41,"stage":"idle","operation_id":null}}}

The application version above is only an example. Clients should check the negotiated protocol version and capabilities instead of assuming support from the Handy version.

Public methods

Method	Purpose
handy.initialize	Negotiate the protocol, identify the client, subscribe to events, and receive the initial state
state.get	Get Handy's current real state
recording.start	Start recording if Handy is idle
recording.stop	Stop the active recording and begin transcription
recording.cancel	Cancel the current recording or processing operation
transcript.get	Fetch one completed transcript by the ID received in an event
transcript.latest	Fetch the latest completed transcript after connecting or reconnecting

Public events

Event	Purpose
state.changed	Reports an actual recording-pipeline state change
transcription.completed	Reports that a transcript is available and provides its ID
operation.failed	Reports that accepted work later failed

State model

Handy has one global microphone and one global transcription pipeline. IPC clients do not own separate recording sessions.

stateDiagram-v2
    [*] --> Idle
    Idle --> Recording: start accepted
    Recording --> Processing: stop accepted
    Recording --> Idle: cancel
    Processing --> Idle: transcription completed
    Processing --> Idle: failed or cancelled

Loading

All clients observe the same state. If two clients send commands at the same time, the coordinator processes them in arrival order.

An operation ID connects later events to the recording that produced them. It does not give one client exclusive control. A recording started from Handy itself should still receive an operation ID and produce the same events.

Recording command examples

Start a normal recording:

{"jsonrpc":"2.0","id":2,"method":"recording.start","params":{"post_process":false}}

{"jsonrpc":"2.0","id":2,"result":{"operation_id":"01K2DEF","state":"recording","changed":true}}

Start with post-processing:

{"jsonrpc":"2.0","id":3,"method":"recording.start","params":{"post_process":true}}

Starting when Handy is already recording is successful but makes no change:

{"jsonrpc":"2.0","id":3,"result":{"operation_id":"01K2DEF","state":"recording","changed":false}}

Stop the active recording:

{"jsonrpc":"2.0","id":4,"method":"recording.stop","params":{}}

{"jsonrpc":"2.0","id":4,"result":{"operation_id":"01K2DEF","state":"processing","changed":true}}

Cancel the current operation:

{"jsonrpc":"2.0","id":5,"method":"recording.cancel","params":{}}

start and stop should be idempotent, meaning that repeating them is safe:

• Starting while already recording returns changed: false.
• Stopping while already idle returns changed: false.
• Starting while Handy is processing returns a busy error because the requested state was not reached.

Event and transcript examples

When Handy enters processing, subscribed clients receive:

{"jsonrpc":"2.0","method":"state.changed","params":{"revision":43,"stage":"processing","operation_id":"01K2DEF"}}

When the transcript is ready, subscribed clients receive its ID but not the transcript text:

{"jsonrpc":"2.0","method":"transcription.completed","params":{"revision":44,"operation_id":"01K2DEF","transcript_id":123,"post_process_requested":true,"post_process_succeeded":true}}

An interested client can then fetch the transcript:

{"jsonrpc":"2.0","id":6,"method":"transcript.get","params":{"transcript_id":123}}

{"jsonrpc":"2.0","id":6,"result":{"id":123,"created_at":1786500000,"raw_text":"This is the raw transcript.","processed_text":"This is the cleaned transcript.","preferred_text":"This is the cleaned transcript.","post_process_requested":true,"post_process_succeeded":true}}

Keeping transcript text out of the completion event has a few benefits:

• Clients only fetch text when they need it.
• State events stay small.
• A reconnecting client can recover through transcript.latest.
• The transcript response can grow later without changing the event shape.

Complete example flow

sequenceDiagram
    participant Client as Local program
    participant IPC as IPC connection layer
    participant App as Shared Handy actions
    participant Coord as TranscriptionCoordinator
    participant History as HistoryManager

    Client->>IPC: handy.initialize
    IPC-->>Client: version, capabilities, current state
    Client->>IPC: recording.start
    IPC->>App: start recording
    App->>Coord: apply one state change at a time
    Coord-->>App: recording started
    IPC-->>Client: accepted, operation ID
    IPC-->>Client: state.changed: recording

    Client->>IPC: recording.stop
    IPC->>App: stop recording
    App->>Coord: apply one state change at a time
    Coord-->>App: processing started
    IPC-->>Client: accepted, processing
    IPC-->>Client: state.changed: processing

    Coord->>History: save completed transcript
    History-->>IPC: transcript available
    IPC-->>Client: transcription.completed: transcript ID
    Client->>IPC: transcript.get
    IPC->>History: fetch transcript ID
    History-->>IPC: transcript data
    IPC-->>Client: transcript result

Loading

Errors

Use normal JSON-RPC errors with stable Handy error names in error.data.

{"jsonrpc":"2.0","id":7,"error":{"code":-32001,"message":"Handy is busy processing another recording","data":{"name":"busy","state":"processing","operation_id":"01K2DEF"}}}

Suggested application errors:

Name	Meaning
busy	The global pipeline cannot accept the request now
invalid_state	The request does not apply to the current state
not_found	The requested transcript does not exist
unsupported	This Handy build or platform does not support the requested capability
permission_denied	Handy lacks an operating-system permission it needs
operation_failed	Accepted work later failed
internal	An unexpected error occurred

Unexpected internal errors should include a diagnostic ID so a developer can find the matching log entry. They should not expose transcript text, filesystem paths, or other sensitive details.

Events and reconnection

• Events are live and delivered at most once.
• Events are ordered within each client connection.
• Handy does not save a list of old events or replay missed events.
• Every state event includes a revision number that always goes up.
• On reconnect, handy.initialize returns the latest state and revision.
• A client that needs the latest result can call transcript.latest.
• Handy limits how many waiting updates it keeps for a slow client. It may disconnect that client rather than block the application or keep using more memory.

Local trust and safety

This proposal trusts processes already running as the current operating-system user.

It should still prevent accidental exposure outside that user:

• Create Unix sockets in a user-private runtime directory with user-only permissions.
• Set the Windows named pipe permissions so only the current user can connect.
• Do not bind a TCP port.
• Limit message size, connection count, and queued events.
• Do not log transcript contents.

Versioning

• Agree on a protocol major version in handy.initialize.
• List each supported feature in the reply.
• Add optional fields, methods, and events without changing the protocol major version.
• Clients should ignore unknown object fields and events they did not request.
• Removing a field or changing its meaning requires a new protocol major version.
• Unknown methods use the normal JSON-RPC method not found error.

Implementation boundaries

The design should keep three layers separate:

1. Application operations — recording control, state queries, and transcript selection.
2. Protocol — stable JSON-RPC request, response, event, and error types.
3. Transport — Unix socket or Windows named pipe connection handling.

This keeps the public protocol independent from Tauri commands, Rust type names, database rows, and platform-specific transport details.

Suggested implementation order

1. Put the shared Handy actions and current state in one place.
2. Add protocol types and tests without any operating-system transport.
3. Add Unix-socket and Windows named-pipe adapters with the same protocol tests.
4. Add initialization, state queries, recording control, and state events.
5. Add completion events and transcript lookup by ID.
6. Test startup, shutdown, sockets left after a crash, reconnects, clients acting at the same time, slow clients, and idempotent state changes on all three platforms.

I am happy to simplify this, build a small proof to test the uncertain parts, or help implement the direction you prefer. Or I can take a whack at a PR. Or whatever -- I'm not trying to enforce this on your project lol

[cjpais]

Thanks for this, I'll take a deeper look tomorrow most likely, but I think I've largely come to a similar set of things and may open a PR with some scaffolding over the next day or two and after I do so we can probably move the discussion there itself