# web-scraper-rust

A polite, single-binary Rust web scraper that crawls pages, converts HTML to Markdown, and tries its best to turn Office documents into something readable too.

It respects `robots.txt`. It waits between requests. It hashes long URLs into filenames because filesystems have feelings too. It does not do cookies, JavaScript rendering, or anything fancy. If you need a headless browser, this is not your tool.

## What It Does

- Crawls internal links from a starting URL (BFS queue)
- Converts HTML pages to Markdown via `htmd`
- Detects and converts documents (PDF, DOCX, XLSX, PPTX, ODT, RTF, EPUB, CSV...) to Markdown via `anydoc`
- Respects `robots.txt` with wildcard pattern matching
- Deduplicates URLs, normalizes fragments and trailing query params
- Configurable request delay, retry with exponential backoff
- File type filtering via `--include-types` / `--exclude-types`
- Path scoping (stay within a subdirectory) or full-site crawl
- Subdomain crawling opt-in
- Single-page mode (grab one page, don't crawl further)
- Structured error logging to JSONL, kept in a separate `_logs` directory

## Tech Stack

| Thing | Choice | Why |
|---|---|---|
| Language | Rust 2024 | Single static binary. No runtime. No GC. No regrets. |
| HTTP client | reqwest + tokio | Async, fast, well-maintained. |
| HTML parsing | scraper | CSS selectors on server-side HTML. |
| HTML→MD | htmd | Strips nav, footer, script tags. Produces clean Markdown. |
| Document conversion | anydoc | Detects format from bytes, converts to Markdown. Falls back to raw. |
| CLI | clap | Derive macros, typed args, good help text. |
| Logging | env_logger | Simple, sufficient. |

## Usage

```bash
# Scrape a single page, save Markdown
web-scraper https://example.com/page --single

# Crawl entire site, only PDFs and HTML
web-scraper https://example.com --types pdf,html

# Crawl a subdirectory only, exclude images
web-scraper https://example.com/docs --exclude-types png,jpg,gif

# Include subdomains, custom delay
web-scraper https://example.com --subdomains --delay-ms 500
```

Output goes to `./<domain>_scraped/`, logs to `./<domain>_scraped_logs/`.

## CLI Flags

```
START_URL              Starting URL to scrape
-o, --output DIR       Output directory (default: <domain>_scraped)
    --subdomains       Also crawl subdomains of the base host
    --single           Only scrape the start URL, don't crawl
    --no-scope         Crawl any path on the same host
    --no-doc-conversion  Save documents as-is, skip conversion
    --delay-ms N       Delay between requests in ms (default: 1000)
    --types a,b,c      Only save these file types (HTML always crawled)
    --exclude-types a,b  Skip these file types
```

`--types` and `--exclude-types` are mutually exclusive. Both still crawl HTML pages for links regardless.

## Build

```bash
just build      # clippy + fmt + release build
# or
cargo build --release
```

Release profile is optimized for size: `opt-level = "z"`, LTO, single codegen unit, stripped.

## Error Logs

Errors are logged to `scrape_errors.jsonl` in the logs directory. Each entry includes timestamp, URL, error type, message, and HTTP status code if applicable. The log file is cleared at the start of each run.

---

**Built to scrape things. Respects robots.txt. Does not require a DevOps team.**