gau: Pull Every Archived URL for a Domain (GitHub, Scanned)

gau, short for getallurls, asks four archives what they know about a domain and prints every URL they return: the Internet Archive's Wayback Machine, Common Crawl, AlienVault's Open Threat Exchange and URLScan. Filters trim the flood by file extension, status code, MIME type and date range, --subs widens it to subdomains, and --fp collapses URLs that differ only in parameter values. It reads domains from stdin and writes URLs to stdout, so it chains with everything else on this list.

Its value is history. A crawler like katana sees what a site links to today; gau shows what used to be there, including old API paths, forgotten admin pages, retired file uploads and query parameters that still work. It never contacts the target, only the archives, which makes it useful for verification and journalism too: checking what a page said before it was edited or removed.

Corben Leo wrote it in 2020, inspired by Tom Hudson's waybackurls, and it is MIT-licensed with about 5,200 stars and 23 contributors. It is small, at about 1,200 lines of Go, and maintained in bursts: the latest release, v2.2.4, dates from October 2024, though fixes for retries and URLScan pagination were merged on 12 October 2026.

• Repository: github.com/lc/gau
• Licence: MIT (MIT License)
• Language: Go. Stars: 5,150. Forks: 527. Last push: Oct 12, 2026.
• Scan: safe, Oct 28, 2026, commit 4eabace

Who it is for

Security teams and bug bounty hunters looking for forgotten endpoints on their own or in-scope domains, OSINT researchers and journalists checking a site's past pages, and SEO or web teams auditing old URLs before a migration.

Getting started

1. Install with Go (or download a release binary, or run the sxcurity/gau Docker image)

go install github.com/lc/gau/v2/cmd/gau@latest

2. List archived URLs for a domain, skipping images and fonts

gau --blacklist png,jpg,gif,svg,woff example.com

3. Include subdomains, keep only pages from 2023 on, and save to a file

gau --subs --from 202301 --o urls.txt example.com

If you use Oh My Zsh, its git plugin already defines gau as an alias for git add --update; run unalias gau or call the binary by its full path. URLScan results are fuller with a free API key in ~/.gau.toml.

Safety scan

We cloned lc/gau at commit 4eabace on Oct 28, 2026 and ran the checks described on the GitHub Tools page: credential patterns, decode-and-execute code, install-time scripts, committed binaries, risky CI workflows, every host the code talks to, known vulnerabilities in pinned dependencies, and project hygiene. A person read every hit. This is what we found.

• No secrets, no pattern hits, no install hooks and no committed binaries in 25 files and about 1,500 lines. The only network hosts in the code are the four providers: web.archive.org, index.commoncrawl.org, otx.alienvault.com and urlscan.io.
• go.mod targets Go 1.20 and pins 26 packages with 28 advisories: 3 high, 4 moderate, the rest unrated. The highs are a logrus denial of service in Entry.Writer, which gau does not call; a fasthttp path traversal in its file server, while gau uses fasthttp only as a client; and klauspost/compress's S2 format. Most of the rest are in golang.org/x/net 0.17.0. Low practical risk, but a sign the project is lightly maintained.
• Two workflows, one building release binaries and one pushing the Docker image, neither using pull_request_target; the 5 third-party action uses are pinned to tags, not commits. Scorecard finds signed release artifacts, which is unusual for a tool this size.
• No security policy, Dependabot, CodeQL or contributing guide; the OpenSSF Scorecard is 4.4. Issues are answered slowly, with 23 open.

What the scanner counted

The raw findings

Every hit the scanner wrote out, with a link to the exact line at the scanned commit. Secrets candidates are redacted.

By the numbers

Lines by language: Go 1,219, Markdown 114, YAML 109, TOML 20.

Questions

Is gau free?

Yes. gau is MIT-licensed and free for any use. The archives it reads are free too, though URLScan limits keyless requests, and the README asks heavy users to consider donating to the Internet Archive and Common Crawl.

Is it legal to use gau on a domain I do not own?

gau only reads public archives and never touches the domain itself, so listing archived URLs is generally lawful. Requesting those URLs from the live site, or testing the parameters you find, is a different matter: do that only on your own sites or with the owner's permission.

How is gau different from waybackurls?

waybackurls, by Tom Hudson, asks only the Wayback Machine. gau adds Common Crawl, AlienVault OTX and URLScan, plus filters for status codes, MIME types, dates and parameters, and a config file for defaults.

This post is part of GitHub Tools, where every repository is cloned and scanned before it is written up. The scan is a snapshot of one commit on one day; the repository has moved on since, so check it before you install.