Crawl4AI is a Python library that drives a real browser through Playwright and hands back Markdown shaped for language models: headings, tables and code kept, navigation and boilerplate filtered out, links turned into citations. Around that it offers structured extraction with CSS or XPath schemas or an LLM of your choice, breadth-first and adaptive deep crawls, sessions, proxies and cookies, a crwl command line, and a self-hosted Docker server with a REST API, a dashboard and MCP support.
It went viral in 2024 as the free, no-account alternative to hosted scrapers and now has about 84,000 stars, still led by its creator, who goes by unclecode. The licence is Apache-2.0 with an added attribution requirement, and a paid Crawl4AI Cloud launched in 2026 for people who would rather not run browsers and proxies themselves. The library remains free to run yourself.
- Repository: github.com/unclecode/crawl4ai
- Licence: Apache-2.0 (Apache License 2.0)
- Language: Python. Stars: 84.3K. Forks: 8,721. Last push: Sep 25, 2026.
- Scan: safe, Sep 25, 2026, commit e5d2e78
Who it is for
Python developers feeding web content into RAG, agents or datasets who want the crawler inside their own code, with full control over the browser, and no per-page bill.
Getting started
1. Install the library
pip install -U crawl4ai2. Install the browser it drives (once)
crawl4ai-setup3. Check the installation
crawl4ai-doctor4. Fetch a page as Markdown from the command line
crwl https://news.ycombinator.com -o markdowncrawl4ai-setup runs playwright install --with-deps, which on Linux installs system libraries and may prompt for sudo. For the Docker server, the README generates a token first and passes it in: the server refuses outside requests without CRAWL4AI_API_TOKEN set.
Safety scan
We cloned unclecode/crawl4ai at commit e5d2e78 on Sep 25, 2026 and ran the checks described on the GitHub Tools page: credential patterns, decode-and-execute code, install-time scripts, committed binaries, risky CI workflows, every host the code talks to, known vulnerabilities in pinned dependencies, and project hygiene. A person read every hit. This is what we found.
- No secrets, no committed binaries and no install hooks across about 287,000 lines. The one pattern hit is webhook.site in the Docker server's security tests, used as a valid external URL for the webhook validator; the bare-IP URLs are test fixtures using documentation addresses.
- SECURITY.md lists what the Docker API server had to fix in 2026: remote code execution through request-supplied hook code and eval (0.8.0, 0.8.1, 0.8.7), a hard-coded default JWT secret, arbitrary file writes and a series of SSRF bypasses. Version 0.9.0 in June made the server secure by default, with authentication on, a loopback bind unless a token is set and hook code removed, and 0.9.4 (September) fixed three more. Every one is documented and fixed, but do not run an old image, and do not expose the server without a token.
- In March 2026 (0.8.6) the litellm dependency was swapped for unclecode-litellm==1.81.13 after a supply-chain compromise of litellm on PyPI. It is a pinned copy published by the maintainer, so you are trusting his account for that package. uv.lock was not updated with it: it still pins litellm 1.74.0.post1, which is where three critical LiteLLM advisories come from, and those concern LiteLLM's proxy server rather than library use.
- 210 known advisories in total. uv.lock (126 packages) has 9 critical and 76 high, led by NLTK 3.9.1's pickle and zip-slip issues; pip installs of the library get a newer NLTK because pyproject.toml asks only for 3.9.1 or later. The Docker server's requirements.txt pins anyio 4.9.0 (1 critical, TLS host-name encoding) and PyJWT 2.10.1 (3 high).
- Four workflows, none using pull_request_target. None of the five third-party actions is pinned to a commit, and one Discord notifier tracks a branch (@master). Security policy, contributing guide and code of conduct present; no Dependabot or CodeQL.
What the scanner counted
| Check | Result |
|---|---|
| Secrets | None found. |
| Suspicious code | 1 pattern hit found and read; every one is listed under the raw findings. |
| Install-time code | None: nothing runs at install beyond the package manager itself. |
| Committed binaries | None. |
| CI workflows | 4 workflows. None use pull_request_target. 5 of 5 third-party actions pinned to a tag rather than a commit. |
| Network hosts | 40 distinct hosts referenced from source; most often httpbin.org, www.nbcnews.com, github.com, docs.python.org. 4 URLs to a bare IP address, listed under the raw findings. |
| Known vulnerabilities | 210 advisories across 131 pinned packages: 9 critical, 79 high, 89 moderate, 33 low. deploy/docker/requirements.txt: 6 packages, 9 advisories; requirements.txt: 1 packages, 0 advisories; uv.lock: 126 packages, 203 advisories. |
| Project hygiene | Has security policy, licence file, contributing guide. Missing automated dependency updates, CodeQL. |
| OpenSSF Scorecard | Not scored: the project is not in Scorecard's weekly index. |
The raw findings
Every hit the scanner wrote out, with a link to the exact line at the scanned commit. Secrets candidates are redacted.
Pattern hits (1)
| Where | Rule | Match |
|---|---|---|
| deploy/docker/tests/test_security_2026_04.py:283 | exfil-host (test/example) | validate_webhook_url("https://webhook.site/test-uuid") |
URLs to bare IP addresses (4)
| Where | Rule | Match |
|---|---|---|
| deploy/docker/tests/test_security_egress_proxy.py:243 | ip-literal-url | assert sent.startswith(b"GET http://203.0.113.7:80/ HTTP/1.1\r\n") |
| tests/memory/test_docker_config_gen.py:27 | ip-literal-url | "BrowserConfig(browser_mode='builtin', proxy_config={'server': 'http://1.2.3.4:8080'})", |
| tests/regression/test_reg_config.py:462 | ip-literal-url | assert pc.server == "http://1.2.3.4:8080" |
| tests/regression/test_reg_config.py:469 | ip-literal-url | assert pc.server == "http://1.2.3.4:8080" |
Worst known vulnerabilities (24 of 210)
| Advisory | Severity | Package | Summary |
|---|---|---|---|
| GHSA-82r6-8w77-94w6 | critical | anyio@4.9.0 | AnyIO: TLSStream IDNA 2003 host name encoding enables potential TLS certificate spoofing |
| GHSA-4xpc-pv4p-pm3w | critical | litellm@1.74.0.post1 | LiteLLM: Authentication Bypass via Host Header Injection |
| GHSA-6wvf-77m9-58rm | critical | litellm@1.74.0.post1 | LiteLLM vulnerable to server-side template injection in the /prompts/test endpoint |
| GHSA-jjhc-v7c2-5hh6 | critical | litellm@1.74.0.post1 | LiteLLM: Authentication bypass via OIDC userinfo cache key collision |
| GHSA-7p94-766c-hgjp | critical | nltk@3.9.1 | NLTK has a Zip Slip Vulnerability |
| GHSA-m4rf-3fr8-xwx3 | critical | nltk@3.9.1 | NLTK: JVM argument injection bypass via per-call options in the NLTK Stanford wrappers (incomplete fix of CVE-2026-12841… |
| GHSA-rhp5-r9x4-f5g2 | critical | nltk@3.9.1 | NLTK: Unsafe Pickle Deserialization in TransitionParser Allows Remote Code Execution |
| GHSA-x99w-6fgc-pmfw | critical | nltk@3.9.1 | NLTK: Allowlisted pickle loaders still permit code execution in current source |
| CVE-2026-0848 | critical | nltk@3.9.1 | |
| GHSA-752w-5fwx-jx9f | high | pyjwt@2.10.1 | PyJWT accepts unknown `crit` header extensions |
| GHSA-xgmm-8j9v-c9wx | high | pyjwt@2.10.1 | PyJWT: Public-key JWK accepted as HMAC secret enables forged HS256 tokens when mixed families are allowed |
| CVE-2025-45768 | high | pyjwt@2.10.1 | |
| GHSA-6mq8-rvhq-8wgg | high | aiohttp@3.12.13 | AIOHTTP's HTTP Parser auto_decompress feature is vulnerable to zip bomb |
| GHSA-cq5v-8q36-5273 | high | aiohttp@3.12.13 | AIOHTTP: Out-of-bounds heap read in C HTTP response parser error path (malformed chunked response) |
| GHSA-2qfp-q593-8484 | high | brotli@1.1.0 | Scrapy is vulnerable to a denial of service (DoS) attack due to flaws in brotli decompression implementation |
| GHSA-47fr-3ffg-hgmw | high | click@8.1.8 | |
| GHSA-537c-gmf6-5ccf | high | cryptography@46.0.3 | Vulnerable OpenSSL included in cryptography wheels |
| GHSA-g6cj-pr64-35w5 | high | cryptography@46.0.3 | cryptography: PKCS#7 EnvelopedData decryption exposes a Bleichenbacher oracle through distinguishable errors and timing |
| GHSA-jwv3-5hgf-82ww | high | cryptography@46.0.3 | python-cryptography: Duplicate self-signed intermediates can cause exponential path-building |
| GHSA-r6ph-v2qm-q3c2 | high | cryptography@46.0.3 | cryptography Vulnerable to a Subgroup Attack Due to Missing Subgroup Validation for SECT Curves |
| GHSA-53mr-6c8q-9789 | high | litellm@1.74.0.post1 | LiteLLM: Privilege escalation via unrestricted proxy configuration endpoint |
| GHSA-69x8-hrgq-fjj8 | high | litellm@1.74.0.post1 | LiteLLM: Password hash exposure and pass-the-hash authentication bypass |
| GHSA-7488-6r32-c95q | high | litellm@1.74.0.post1 | LiteLLM: MCP Authentication Bypass via OAuth2 Passthrough Fallback |
| GHSA-qrc4-49gv-mv9m | high | litellm@1.74.0.post1 | LiteLLM allows an authenticated internal_user to create API keys with access to routes that their role does not permit |
By the numbers
| Stars | 84.3K |
|---|---|
| Forks | 8,721 |
| Contributors | 95 |
| Commits | 1,697 |
| Open issues | 39 |
| Open pull requests | 168 |
| Releases | 22 |
| Latest release | v0.9.4 |
| Licence | Apache-2.0 |
| Main language | Python |
| Project age | 2 years |
| Last push | Sep 25, 2026 |
| Tracked files | 943 |
| Lines of code | 287.3K |
| Checkout size | 29 MB |
Lines by language: Python 149.7K, Markdown 75.4K, JavaScript 22.1K, CSS 16.4K, HTML 11K, Jupyter 10.8K.
Questions
Is Crawl4AI free?
Yes, to run yourself. The library, CLI and Docker server are Apache-2.0, but the LICENSE file adds an attribution requirement: public uses and derivative works must credit Crawl4AI (the README offers badges for this). Crawl4AI Cloud is the paid option, pay as you go, with the first $10 free until 31 December 2026 and a $5 minimum after that.
Does Crawl4AI need an LLM or API key?
No. Crawling, Markdown generation, content filtering and CSS or XPath extraction all run locally without any key. An LLM is only needed for LLM-based extraction or the crwl question feature, and it goes through LiteLLM, so OpenAI, Anthropic, Gemini or a local model through Ollama all work. The cloud service needs its own key.
Can Crawl4AI handle JavaScript-heavy sites?
Yes. It runs a real Chromium through Playwright, so pages render as in a browser, and it can execute your own JavaScript, wait for elements, keep sessions and use proxies. It also ships patchright and playwright-stealth for sites that block automation. Sites with aggressive bot protection may still need proxies, which is what the paid cloud handles for you.
This post is part of GitHub Tools, where every repository is cloned and scanned before it is written up. The scan is a snapshot of one commit on one day; the repository has moved on since, so check it before you install.