Firecrawl: Turn Any Website into LLM-Ready Data (GitHub, Scanned)

Firecrawl turns web pages into something a language model can use. Give it a URL and it returns clean Markdown, HTML, structured JSON or a screenshot, with JavaScript rendered and boilerplate stripped; give it a site and it crawls or maps every page; give it a query and it searches the web and returns the full content of the results. It can click, scroll and type before extracting, parses PDFs and DOCX files, and has an agent endpoint that gathers data from a plain-language request.

It is the most-starred project in its category, at about 185,000 stars, built by the company behind Mendable and offered two ways. The hosted API at firecrawl.dev handles proxies and bot walls and adds features the open-source build lacks; this repository is the same API, AGPL-3.0, that you can run yourself with Docker Compose. SDKs for Python, Node, Go, Java, Rust and PHP are MIT-licensed, and an MCP server and CLI plug it into coding agents.

• Repository: github.com/firecrawl/firecrawl
• Licence: AGPL-3.0 (GNU Affero General Public License v3.0)
• Language: TypeScript. Stars: 185.2K. Forks: 9,935. Last push: Sep 27, 2026.
• Scan: safe, Sep 25, 2026, commit f75a8d4

Who it is for

Developers building RAG pipelines, research agents or monitoring tools that need web content as clean text, and teams that want the same API self-hosted for cost or data-control reasons.

Getting started

1. Python SDK, for the hosted API (get a key at firecrawl.dev) or your own instance

pip install firecrawl-py

2. Or the Node SDK

npm install firecrawl

3. Self-host: clone and start the Docker Compose stack

git clone https://github.com/firecrawl/firecrawl.git && cd firecrawl && docker compose up -d

4. Scrape a page through your own instance

curl -X POST http://localhost:3002/v2/scrape -H 'Content-Type: application/json' -d '{"url": "firecrawl.dev"}'

The self-hosted API needs no key because it has no authentication by default. The project's own SELF_HOST.md says to keep it on a trusted network until you add authentication, TLS and network policy, and notes the Compose file defines no persistent volumes.

Safety scan

We cloned firecrawl/firecrawl at commit f75a8d4 on Sep 25, 2026 and ran the checks described on the GitHub Tools page: credential patterns, decode-and-execute code, install-time scripts, committed binaries, risky CI workflows, every host the code talks to, known vulnerabilities in pinned dependencies, and project hygiene. A person read every hit. This is what we found.

• One secret hit: a private key in apps/api/src/__tests__/snips/v2/scrape-skip-tls.test.ts, named SELF_SIGNED_KEY. The test starts an HTTPS server with a deliberately bad certificate to check the skipTlsVerification option; the key protects nothing. The bare-IP URLs are also tests, including phishing-style examples for the URL canonicalizer.
• Three pattern hits. apps/api/Dockerfile installs Rust with rustup's official curl | sh at image build time, on the build machine. The other two are in tests and examples: a Java SDK test that uses webhook.site as a placeholder, and a blog notebook quoting Heroku's installer.
• Install hooks matter only if you build from source: apps/api/native's install script runs pnpm build to compile a Rust module (with its build.rs), and apps/api's prepare script sets up husky git hooks for contributors. One committed binary, the Java SDK's Gradle wrapper jar.
• 48 known advisories and none critical. The API itself (apps/api/pnpm-lock.yaml, 930 packages) has 1 high and 1 moderate. Most of the rest sit in an example project (9 high), the Rust SDK's Cargo.lock (1 high, rustls-webpki) and unrated entries in the Go HTML-to-Markdown service. Dependabot is configured.
• 31 workflows, none using pull_request_target, and 33 of 41 third-party actions pinned to a commit, one of the best ratios in this series. Licence and contributing guide present; no security policy file or CodeQL.

What the scanner counted

The raw findings

Every hit the scanner wrote out, with a link to the exact line at the scanned commit. Secrets candidates are redacted.

By the numbers

Lines by language: TypeScript 263.3K, Python 49.3K, Markdown 15.7K, JSON 14.3K, Rust 12.1K, Jupyter 9,099.

Questions

Is Firecrawl free?

The code is. The server is AGPL-3.0 and free to self-host, and the SDKs are MIT. The hosted API at firecrawl.dev is paid by credits, with a free allowance to start, and includes extras the open-source build lacks, such as its own scraping engine and proxy handling. If you modify the server and offer it to others, the AGPL requires sharing your changes.

Firecrawl or Crawl4AI?

Firecrawl is an API service first: you call endpoints from any language, and self-hosting means running a multi-container stack with Redis, RabbitMQ and PostgreSQL. Crawl4AI is a Python library first that runs a browser inside your own process, with a Docker server as an option. For a hosted API with SDKs, Firecrawl; for crawling inside a Python program, Crawl4AI.

Does Firecrawl respect robots.txt?

By default, yes. The crawler reads each site's robots.txt and skips disallowed URLs, and the code offers an ignoreRobotsTxt option for crawls that override it. The README is explicit that respecting each site's policies and terms is the user's responsibility, which applies equally to a self-hosted instance.

This post is part of GitHub Tools, where every repository is cloned and scanned before it is written up. The scan is a snapshot of one commit on one day; the repository has moved on since, so check it before you install.