nautogui

English | 中文

Cross-platform GUI automation for Node.js: mouse, keyboard, screenshot, image template locate. No LLM.

Backend	Purpose
@nut-tree/nut-js	OS mouse / keyboard / screen
sharp	locate() / locateAll() in haystack images

Install

From npm (after publish)

npm install nautogui

Dependencies @nut-tree/nut-js and sharp are installed automatically with the package.

Local package (development / before publish)

Source lives in nautogui/. Build first (tsc → dist/) before use.

Option A: Monorepo reference (default in this repo)

In the root package.json:

"dependencies": {
  "nautogui": "file:nautogui"
}

# repo root
cd nautogui
npm install
npm run build

cd ..\..
npm install

Then any subproject can:

import * as nautogui from "nautogui";

Root npm run build:tsc / npm test runs build:nautogui to compile the local package first.

Option B: Install from a local path in another project

npm install D:\WorkSpace\nautogui
# or relative path
npm install ../nautogui

Your package.json will contain something like:

"nautogui": "file:../nautogui"

After editing nautogui source, rebuild:

cd nautogui
npm run build

Option C: Pack as .tgz and install

cd nautogui
npm install
npm run build
npm pack
# creates nautogui-0.3.0.tgz

cd ..\your-other-project
npm install D:\WorkSpace\nautogui\nautogui-0.3.0.tgz

Option D: Global link (local integration testing)

cd nautogui
npm install
npm run build
npm link

cd ..\your-other-project
npm link nautogui

To unlink: npm unlink nautogui (in your project) and npm unlink -g nautogui (in the nautogui directory).

CLI (optional)

After build and install:

npx nautogui-mouseinfo

Shows live mouse coordinates (floating window on Windows, terminal elsewhere).

Quick start

All API calls are async.

Keyboard and mouse

import * as nautogui from "nautogui";

const { width: screenWidth, height: screenHeight } = await nautogui.size();
const { x: currentMouseX, y: currentMouseY } = await nautogui.position();

await nautogui.moveTo(100, 150);
await nautogui.click();
await nautogui.click(200, 220);
await nautogui.move(0, 10); // relative: 10px down
await nautogui.doubleClick();
await nautogui.moveTo(500, 500, { duration: 2, tween: nautogui.easeInOutQuad });
await nautogui.write("Hello world!", { interval: 0.25 });
await nautogui.press("esc");
await nautogui.keyDown("shift");
await nautogui.write(["left", "left", "left", "left", "left", "left"]);
await nautogui.keyUp("shift");
await nautogui.hotkey("ctrl", "c");

Screenshots

import * as nautogui from "nautogui";

const path1 = await nautogui.screenshot();
const path2 = await nautogui.screenshot("my_screenshot2.png");
await nautogui.screenshot("region.png", { left: 0, top: 0, width: 400, height: 300 });

Image template locate

import * as nautogui from "nautogui";

const button7location = await nautogui.locateOnScreen("button.png");
// → { left: 1416, top: 562, width: 50, height: 41 }

if (button7location) {
  const { x: buttonx, y: buttony } = nautogui.center(button7location);
  await nautogui.click(buttonx, buttony);
}

const centerPt = await nautogui.locateCenterOnScreen("button.png");
if (centerPt) {
  await nautogui.click(centerPt.x, centerPt.y);
}

const box = await nautogui.locate("button.png", "haystack.png");

Message boxes

await nautogui.alert("This is an alert box.");
await nautogui.confirm("Shall I proceed?");
await nautogui.confirm("Enter option.", { buttons: ["A", "B", "C"] });
await nautogui.prompt("What is your name?");
await nautogui.password("Enter password (text will be hidden)");

Runnable examples

cd nautogui
npm run build
npm run fixtures
node examples/01-keyboard-mouse-control.mjs          # dry-run
node examples/01-keyboard-mouse-control.mjs --live   # real mouse/keyboard

See examples/README.md for all example scripts.

Design

• Operations use screen coordinates (x, y) or PNG template images
• locateOnScreen('button.png') uses nut.js template matching
• No LLM / natural-language UI locating

For “describe UI → find coordinates”, wire a vision model in your host app (e.g. this repo’s browser/desktop vision + Stagehand Agent), then call nautogui.click(x, y).

Build & pack

cd nautogui
npm install && npm run build && npm pack && npm test

Notes

1. All functions are async (await).
2. msgbox uses native OS dialogs when available, else readline.
3. Primary monitor only.

Apache-2.0