Scrape a Website Safely with a CLI

---
name: scrape-website-workflow
description: "Define a small URL scope, choose extraction or browser automation, and preserve each source URL with the result."
---

# Extract content from a website

## Intended outcome

Collect the specific public content needed for a task without uncontrolled crawling or hidden interaction.

## Evidence boundary

- Tool status distinguishes `docs-verified` material from independent execution; the two are not interchangeable.
- The current registry does not attribute a tool execution to a particular agent, so compatible agents cannot be called tested agents.
- Reconfirm the target account, environment, version, and official documentation before execution.
- Never auto-execute an R2, R3, or confirmation-marked step; obtain explicit approval immediately before it runs.

## Recommended tools, installation, and authentication

- **Firecrawl CLI** (evidence: `docs-verified`, docs checked: `2026-07-10`, no independently executed version recorded)
  - Install: `npx firecrawl --help`
  - Authentication: API key
  - Minimum permissions: Use a dedicated key with quota limits and crawl only authorized targets.
- **Playwright** (evidence: `docs-verified`, docs checked: `2026-07-10`, no independently executed version recorded)
  - Install: `npm install --save-dev @playwright/test`
  - Authentication: not required for the base operation
  - Minimum permissions: No service credential is required; restrict filesystem and network access to the task.

## Input contract

- Allowed URLs
- required fields or page state
- crawl and rate limits
- site access constraints

## Output contract

- Extracted text or structured fields
- source URL ledger
- failure list
- screenshots or traces when needed

## Safe workflow

1. **Set the boundary** — List allowed hosts and URLs, excluded areas, maximum pages, and expected output.
   - Input: Research question and site scope
   - Output: Bounded collection plan
   - Risk: `read-only`
2. **Choose the interface** — Use extraction for readable content; use Playwright only when rendering or interaction is necessary.
   - Input: Page behavior and output need
   - Output: Selected tool and collection commands
   - Risk: `read-only`
3. **Collect and cite** — Pace requests, save source URLs and timestamps, and report blocked or incomplete pages.
   - Input: Approved plan
   - Output: Content packet with provenance
   - Risk: `local-write`

## Ask before proceeding

- Logging in, accepting terms, submitting a form, or changing remote state
- Expanding beyond the agreed host or page count
- Collecting personal, restricted, or paywalled information

## Rollback

- Stop the collection immediately when scope changes
- Delete local captures that should not have been collected
- Do not attempt to undo remote side effects through more automation

## Official sources

- [Firecrawl documentation](https://docs.firecrawl.dev/) — Official extraction and crawl API documentation, formats, and usage boundaries.
- [Playwright documentation](https://playwright.dev/docs/intro) — Official browser automation, test runner, trace, and screenshot documentation.