discovered 30 Mar 2026

urlfinder

Go ★ 855 via github-topic

AI Summary: URLFinder is a high-speed, passive URL discovery tool optimized for efficient web asset discovery without active scanning, making it particularly useful for penetration testers and security researchers. It features curated passive sources for comprehensive URL gathering, supports multiple output formats, and provides integration capabilities through STDIN/OUT support.

README

URLFinder

A high-speed tool for passively gathering URLs, optimized for efficient web asset discovery without active scanning.

Features • Installation • Usage • Examples • Join Discord

Overview

URLFinder is a high-speed, passive URL discovery tool designed to simplify and accelerate web asset discovery, ideal for penetration testers, security researchers, and developers looking to gather URLs without active scanning.

Features

Curated Passive Sources to maximize comprehensive URL discovery
Supports multiple output formats (JSON, file, stdout)
Optimized for Speed and resource efficiency
STDIN/OUT support for easy integration into existing workflows

Installation

URLFinder requires Go 1.21. Install it using the following command or download a pre-compiled binary from the releases page.

go install -v github.com/projectdiscovery/urlfinder/cmd/urlfinder@latest

Usage

urlfinder -h

This command displays help for URLFinder. Below are some common switches and options.

A streamlined tool for discovering associated URLs.

Usage:
  ./urlfinder [flags]

Flags:
INPUT:
   -d, -list string[]  target domain or list of domains

SOURCE:
   -s, -sources string[]           specific sources for discovery (e.g., -s alienvault,commoncrawl)
   -es, -exclude-sources string[]   sources to exclude (e.g., -es alienvault,commoncrawl)
   -all                             use all sources (may be slower)
SCOPE:
   -us, -url-scope string[]       in scope url regex to be followed by urlfinder
   -uos, -url-out-scope string[]  out of scope url regex to be excluded by urlfinder
   -fs, -field-scope string       pre-defined scope field (dn,rdn,fqdn) or custom regex (e.g., '(company-staging.io|company.com)') (default "rdn")
   -ns, -no-scope                 disables host based default scope
   -do, -display-out-scope        display external endpoint from scoped crawling

FILTER:
   -m, -match string[]       URLs or list to match (file or comma-separated)
   -f, -filter string[]      URLs or list to filter (file or comma-separated)

RATE-LIMIT:
   -rl, -rate-limit int      max HTTP requests per second (global)
   -rls, -rate-limits value  per-provider HTTP request limits (e.g., -rls waybackarchive=15/m)

UPDATE:
   -up, -update              update URLFinder to the latest version
   -duc, -disable-update-check  disable automatic update checks

OUTPUT:
   -o, -output string       specify output file
   -j, -jsonl               JSONL output format
   -od, -output-dir string  specify output directory
   -cs, -collect-sources    include all sources in JSON output

CONFIGURATION:
   -config string           config file (default "$CONFIG/urlfinder/config.yaml")
   -pc, -provider-config string  provider config file (default "$CONFIG/urlfinder/provider-config.yaml")
   -proxy string            HTTP proxy

DEBUG:
   -silent                  show only URLs in output
   -version                 display URLFinder version
   -v                       verbose output
   -nc, -no-color           disable colored output
   -ls, -list-sources       list all available sources
   -stats                   display source statistics

OPTIMIZATION:
   -timeout int   timeout in seconds (default 30)
   -max-time int  max time in minutes for enumeration (default 10)

Examples

Basic Usage

urlfinder -d tesla.com

This command enumerates URLs for the target domain tesla.com.

Example run:

$ urlfinder -d tesla.com

  __  _____  __   _____         __       
 / / / / _ \/ /  / __(_)__  ___/ /__ ____
/ /_/ / , _/ /__/ _// / _ \/ _  / -_) __/
\____/_/|_/____/_/ /_/_//_/\_,_/\__/_/    										

		projectdiscovery.io

[INF] Current urlfinder version v0.0.1 (latest)
[INF] Enumerating urls for tesla.com
https://www.tesla.com/akam/13/7e68a6e8
https://www.tesla.com/akam/13/pixel_4e07b670
https://www.tesla.com/da_dk/en/node/30788?redirect=no
https://www.tesla.com/de_at/findus/location/charger/dc6290
https://www.tesla.com/akam/13/7ade0a44
https://www.tesla.com/cs_cz/referral/teslaapp23713?redirect=no
https://www.tesla.com/da_dk/findus/location/charger/dc253
https://www.tesla.com/akam/13/pixel_76102729
https://www.tesla.com/da_dk/blog/modules//system/system.messages.js
...
[INF] Found 202435 urls for tesla.com in 2 minutes 37 seconds

Filtering Options

Use the -m (match) and -f (filter) options to refine results based on URL patterns.

Examples

Include URLs Matching Specific Patterns
To include only URLs containing “shop” or “model”:
```
urlfinder -d tesla.com -m shop,model
```
Exclude URLs Matching Specific Patterns
To exclude URLs containing “privacy” or “terms”:
```
urlfinder -d tesla.com -f privacy,terms
```
Combined Match and Filter
To find URLs containing “support” but exclude those with “faq”:
```
urlfinder -d tesla.com -m support -f faq
```

Using Files for Matching and Filtering

Provide patterns in files:

urlfinder -d tesla.com -m include-patterns.txt -f exclude-patterns.txt

JSONL Output Example

Use the -j or --jsonl flag to output results in JSONL (JSON Lines) format, where each line is a separate JSON object. This format is useful for processing large outputs in a structured way.

Command Example

urlfinder -d tesla.com -j

Example JSONL Output

{"url":"https://shop.tesla.com/product/model-s-plaid","input":"tesla.com","source":"waybackarchive"}
{"url":"https://www.tesla.com/inventory/used/ms","input":"tesla.com","source":"waybackarchive"}
{"url":"https://forums.tesla.com/discussion/101112/model-3-updates","input":"tesla.com","source":"waybackarchive"}

Each JSON object contains:

url: The discovered URL.
input: The target domain (e.g., tesla.com).
source: The data source for the URL discovery (e.g., waybackarchive).

URLFinder is made with ❤️ by the ProjectDiscovery team and distributed under the MIT License.