Midscene.js lets you describe UI actions and checks in plain language and has a vision model carry them out. Instead of CSS selectors you write 'search for headphones and filter to under $100', then assert 'every result costs less than $100', and Midscene works from screenshots to click, type, scroll and verify. That means it can handle canvas elements, icon-only buttons, cross-origin iframes and native apps that selector-based tools struggle with.
The same API covers web (through Playwright, Puppeteer or your desktop Chrome), Android, iOS, HarmonyOS and desktop apps on macOS, Windows and Linux. Tests can be written as YAML cases with Midscene Test or added to existing Playwright code, and every run produces an HTML report with screenshots and the model's reasoning. It works with a range of vision models, including open ones you can host yourself.
It comes from the web-infra-dev organisation, the team behind Rspack and Rsbuild, and is MIT-licensed with about 15,400 stars and 90 contributors, at v1.17.1 when scanned. Each step is a model call, so tests are slower and cost more than classic selectors, and results depend on the model you choose.
- Repository: github.com/web-infra-dev/midscene
- Licence: MIT (MIT License)
- Language: TypeScript. Stars: 15.4K. Forks: 1,206. Last push: Oct 26, 2026.
- Scan: safe, Oct 26, 2026, commit 56bdbd0
Who it is for
QA engineers and developers whose selector-based tests keep breaking, teams testing mobile or desktop apps as well as the web, and anyone automating a UI they do not control.
Getting started
1. Create a Midscene Test project (needs Node.js 20.19+ and pnpm)
pnpm dlx @midscene/test create2. Or add it to an existing Playwright project
npm install @midscene/web dotenv --save-devConfigure a vision model first, with an API key or a self-hosted endpoint, as described in the model setup guide on midscenejs.com. To try it without code, install the Midscene Chrome extension from the quick start page.
Safety scan
We cloned web-infra-dev/midscene at commit 56bdbd0 on Oct 26, 2026 and ran the checks described on the GitHub Tools page: credential patterns, decode-and-execute code, install-time scripts, committed binaries, risky CI workflows, every host the code talks to, known vulnerabilities in pinned dependencies, and project hygiene. A person read every hit. This is what we found.
- No secrets. The 8 pattern hits are a regular expression that splits base64 image data URLs in tests, which the scanner reads as decode-then-run, and a code comment about PowerShell's -EncodedCommand in packages/computer/src/windows-dpi.ts.
- 3 committed Mach-O binaries in packages/computer/bin/darwin (rdp-helper, 467 KB; display-info and phased-scroll, 50 KB each) are helpers for desktop control on macOS. Their source is in packages/computer/native, though the scan cannot prove the binaries were built from it.
- Install hooks: the example projects run midscene-test nodes after install, and the Android packages download scrcpy-server and YADB from those projects' GitHub releases when Midscene publishes, pinned by version rather than by hash.
- 518 known advisories (19 critical, 246 high) in the monorepo's single pnpm-lock.yaml of 2,448 packages, which covers the documentation site, examples, test apps and build tooling as well as the libraries. Installing @midscene/web resolves its own, smaller dependency tree.
- Screenshots and page content go to the model provider you choose; its data privacy page says Midscene itself receives nothing. An optional Sign in with ChatGPT mode, OpenAI's sanctioned token-sharing flow, stores tokens in ~/.midscene/siwc.json and uses your ChatGPT plan's quota.
- 15 workflows, none using pull_request_target, and all 13 third-party actions are pinned to commits. Licence and contributing guide present; no security policy, Dependabot or CodeQL.
What the scanner counted
| Check | Result |
|---|---|
| Secrets | None found. |
| Suspicious code | 8 pattern hits found and read; every one is listed under the raw findings. |
| Install-time code | 8 npm lifecycle scripts. 2 installer scripts |
| Committed binaries | 3 executable or compiled objects committed; listed under the raw findings. |
| CI workflows | 15 workflows. None use pull_request_target. 0 of 13 third-party actions pinned to a tag rather than a commit. |
| Network hosts | 40 distinct hosts referenced from source; most often github.com, midscenejs.com, gateway.example, todomvc.com. No URLs to bare IP addresses. |
| Known vulnerabilities | 518 advisories across 2,448 pinned packages: 19 critical, 246 high, 215 moderate, 38 low. pnpm-lock.yaml: 2,448 packages, 518 advisories. |
| Project hygiene | Has licence file, contributing guide. Missing security policy, automated dependency updates, CodeQL. |
| OpenSSF Scorecard | Not scored: the project is not in Scorecard's weekly index. |
The raw findings
Every hit the scanner wrote out, with a link to the exact line at the scanned commit. Secrets candidates are redacted.
Pattern hits (8)
| Where | Rule | Match |
|---|---|---|
| packages/android/tests/ai/android-emulator-smoke.test.ts:289 | decode-then-eval (test/example) | const match = /^data:image\/\w+;base64,(.+)$/s.exec(base64); |
| packages/android/tests/ai/todo.test.ts:55 | decode-then-eval (test/example) | const match = /^data:image\/\w+;base64,(.+)$/s.exec(base64); |
| packages/computer/src/windows-dpi.ts:87 | powershell-encoded | * Node, `powershell.exe -EncodedCommand` exited with status 0 but produced no |
| packages/computer/tests/ai/ai-auto-todo.test.ts:44 | decode-then-eval (test/example) | const match = /^data:image\/\w+;base64,(.+)$/s.exec(base64); |
| packages/computer/tests/ai/macos-desktop-smoke.test.ts:355 | decode-then-eval (test/example) | const match = /^data:image\/\w+;base64,(.+)$/s.exec(base64); |
| packages/ios/tests/ai/ios-app-switcher.test.ts:24 | decode-then-eval (test/example) | const match = /^data:image\/\w+;base64,(.+)$/s.exec(base64); |
| packages/ios/tests/ai/ios-simulator-smoke.test.ts:87 | decode-then-eval (test/example) | const match = /^data:image\/\w+;base64,(.+)$/s.exec(base64); |
| packages/ios/tests/ai/todo.test.ts:81 | decode-then-eval (test/example) | const match = /^data:image\/\w+;base64,(.+)$/s.exec(base64); |
npm lifecycle scripts (8)
examples/catalog/test/android/package.jsonpostinstall:midscene-test nodesexamples/catalog/test/computer/package.jsonpostinstall:midscene-test nodesexamples/catalog/test/harmony/package.jsonpostinstall:midscene-test nodesexamples/catalog/test/ios/package.jsonpostinstall:midscene-test nodesexamples/catalog/test/web/package.jsonpostinstall:midscene-test nodespackage.jsonprepare:husky && pnpm run buildpackages/android-playground/package.jsonprepack:node ../android/scripts/download-scrcpy-server.mjs --target=bin/scrcpy-serverpackages/android/package.jsonprepack:node scripts/download-scrcpy-server.mjs && node scripts/download-yadb.mjs
Installer scripts (2)
Committed binaries (3)
packages/computer/bin/darwin/rdp-helper: Mach-O, 467 KBpackages/computer/bin/darwin/display-info: Mach-O, 50 KBpackages/computer/bin/darwin/phased-scroll: Mach-O, 50 KB
Worst known vulnerabilities (24 of 518)
| Advisory | Severity | Package | Summary |
|---|---|---|---|
| GHSA-5rq4-664w-9x2c | critical | basic-ftp@5.0.5 | Basic FTP has Path Traversal Vulnerability in its downloadToDir() method |
| GHSA-cpq7-6gpm-g9rc | critical | cipher-base@1.0.4 | cipher-base is missing type checks, leading to hash rewind and passing on crafted data |
| GHSA-vjh7-7g9h-fjfh | critical | elliptic@6.6.0 | Elliptic's private key extraction in ECDSA upon signing a malformed input (e.g. a string) |
| GHSA-fjxv-7rqg-78g4 | critical | form-data@4.0.1 | form-data uses unsafe random function in form-data for choosing boundary |
| GHSA-fjxv-7rqg-78g4 | critical | form-data@4.0.2 | form-data uses unsafe random function in form-data for choosing boundary |
| GHSA-xvch-5gv4-984h | critical | minimist@1.2.5 | Prototype Pollution in minimist |
| GHSA-h7cp-r72f-jxh6 | critical | pbkdf2@3.1.2 | pbkdf2 returns predictable uninitialized/zero-filled memory for non-normalized or unimplemented algos |
| GHSA-v62p-rq8g-8h59 | critical | pbkdf2@3.1.2 | pbkdf2 silently disregards Uint8Array input, returning static keys |
| GHSA-xq3m-2v4x-88gg | critical | protobufjs@8.0.0 | Arbitrary code execution in protobufjs |
| GHSA-jqcg-44mw-7w3h | critical | proxy-addr@2.0.7 | proxy-addr vulnerable to IP spoofing via IPv4-mapped IPv6 trust subnet |
| GHSA-95m3-7q98-8xr5 | critical | sha.js@2.4.11 | sha.js is missing type checks leading to hash rewind and passing on crafted data |
| GHSA-w7jw-789q-3m8p | critical | shell-quote@1.7.3 | shell-quote quote() does not escape newlines in object .op values |
| GHSA-w7jw-789q-3m8p | critical | shell-quote@1.8.2 | shell-quote quote() does not escape newlines in object .op values |
| GHSA-w7jw-789q-3m8p | critical | shell-quote@1.8.3 | shell-quote quote() does not escape newlines in object .op values |
| GHSA-23hp-3jrh-7fpw | critical | tar@6.2.1 | node-tar: Decompression/parse DoS via unlimited input |
| GHSA-23hp-3jrh-7fpw | critical | tar@7.4.3 | node-tar: Decompression/parse DoS via unlimited input |
| GHSA-5gmw-xhrv-c9v3 | critical | tinypool@1.1.1 | Tinypool: Prototype Pollution gadget in worker options leads to Remote Code Execution |
| GHSA-85c8-ppgw-ccpr | critical | tinypool@1.1.1 | Tinypool: Prototype Pollution Gadget to RCE in run() options |
| GHSA-5xrq-8626-4rwp | critical | vitest@3.0.5 | When Vitest UI server is listening, arbitrary file can be read and executed |
| GHSA-jmg2-rcxh-w8q3 | high | @rsdoctor/rspack-plugin@1.0.2 | @rsdoctor/rspack-plugin has Unauthenticated HTTP API that Exposes Project Source Code and Build Metadata |
| GHSA-27p8-2357-5qqv | high | @xmldom/xmldom@0.8.10 | xmldom: DocType `name` Injection Bypasses requireWellFormed |
| GHSA-2v35-w6hq-6mfw | high | @xmldom/xmldom@0.8.10 | xmldom: Uncontrolled recursion in XML serialization leads to DoS |
| GHSA-4w3w-2rp5-g8jm | high | @xmldom/xmldom@0.8.10 | xmldom: Attribute name injection via setAttribute() bypasses requireWellFormed |
| GHSA-8344-3jmq-59r6 | high | @xmldom/xmldom@0.8.10 | xmldom: Quadratic-time attribute deduplication |
By the numbers
| Stars | 15.4K |
|---|---|
| Forks | 1,206 |
| Contributors | 90 |
| Commits | 2,411 |
| Open issues | 44 |
| Open pull requests | 72 |
| Releases | 222 |
| Latest release | v1.17.1 |
| Licence | MIT |
| Main language | TypeScript |
| Project age | 2 years |
| Last push | Oct 26, 2026 |
| Tracked files | 2,931 |
| Lines of code | 496.8K |
| Checkout size | 112 MB |
Lines by language: TypeScript 385K, JSON 41K, Markdown 33.2K, JavaScript 13.1K, CSS 7,884, YAML 5,856.
Questions
Is Midscene.js free?
Yes. Midscene.js is free and open source under the MIT licence. You pay for the vision model it calls, either per token from an API provider, from a ChatGPT plan, or nothing beyond hardware if you host an open model yourself.
Which models work with Midscene?
It needs a vision model that can locate elements in screenshots. The project documents Qwen, Doubao Seed, GLM-4.6V, Gemini Flash, GPT models, DeepSeek and UI-TARS, including open models you can self-host; its model selection guide compares them.
Does Midscene replace Playwright?
No, it works alongside it. In a Playwright test you create a Midscene agent on the page and mix AI steps with normal Playwright code, using natural language where selectors are brittle and plain code where they are not.
This post is part of GitHub Tools, where every repository is cloned and scanned before it is written up. The scan is a snapshot of one commit on one day; the repository has moved on since, so check it before you install.