name: dyagil-firecrawl
description: Scrape, search, map, and crawl the web for AI agents via the Firecrawl API. Use when your agent needs clean markdown from JS-heavy or SPA sites, search results with full page content, site URL mapping, or deep documentation crawls — i.e. anywhere basic HTTP fetching fails.
version: 1.0.0
license: MIT
author: dyagil

Firecrawl Skill — Web Data for Agents

Principle

Firecrawl is a managed API for the three cases where a basic web_fetch agent tool falls short:

1. JS-heavy sites — React/SPA pages that need a real browser to render.
2. Search with full content — standard search APIs return only snippets; Firecrawl returns rendered markdown per result.
3. Map / Crawl — list every URL on a site, mirror a docs tree, etc.

When to Use

| Task | Recommended |
|---|---|
| Plain HTML article | web_fetch (free, fast) |
| Quick search | basic web search tool (free) |
| JS-heavy / SPA page | fc scrape |
| Search + full content | fc search --scrape |
| Full doc-site crawl | fc crawl |
| URL inventory of a site | fc map |
| Page needing clicks / login | fc interact (REST only, see below) |

Rule of thumb: if web_fetch returns empty markup or noise, escalate to fc scrape.

Commands

scrape — single page

fc scrape https://example.com                            # markdown to stdout
fc scrape https://example.com --format html --out a.html
fc scrape https://example.com --out ~/scraped/page.md

search

fc search "best running shoes 2026" --limit 5
fc search "topic" --limit 3 --scrape           # include full content of each hit
fc search "..." --out results.json

map — list URLs on a site

fc map https://docs.example.com --limit 200
fc map https://example.com --out urls.txt

crawl — deep crawl of many pages

fc crawl https://docs.example.com --limit 50 --out ~/docs-mirror/

⚠️ Takes minutes. Each page = 1 credit. Don't blast it on a tight free-tier budget.

ask / docs — debug and help

fc ask "why did my scrape return empty?" --jobId abc123
fc docs "how do I bypass cloudflare?"

status

fc status   # prints key prefix + a smoke-test scrape

Pricing (Remember This)

• Free tier: 500 credits / month.
• Each scrape or search result ≈ 1 credit.
• Each crawl page = 1 credit.
• Past the free tier you pay.

Before large crawl/map calls with high --limit, warn the user about credit cost.

Recommended Patterns

Pattern 1 — Regular article

# Try the free tool first.
# If it returns empty / unreadable:
fc scrape <url> --out /tmp/article.md

Pattern 2 — Research on a topic

fc search "<topic>" --limit 5 --scrape --out /tmp/research.json
# Then parse the JSON and pull what you need.

Pattern 3 — Whole docs site

fc map https://docs.example.com --limit 100    # get the URL list
# Pick relevant URLs and:
fc scrape <selected_urls>...
# Or in bulk:
fc crawl https://docs.example.com --limit 20 --out ~/mirror/

Credentials

• Key: store at ~/.openclaw/credentials/firecrawl/api_key (chmod 600).
• Base URL: https://api.firecrawl.dev/v2
• Auth: Authorization: Bearer fc-...
• CLI: ~/bin/fc → your local fc.cjs (Node CommonJS, zero dependencies).

Get a key at: https://firecrawl.dev (free tier available).

Direct REST (when the CLI isn't enough)

Example — interact (clicks / form-fill) which isn't wrapped in the CLI yet:

KEY=$(cat ~/.openclaw/credentials/firecrawl/api_key)
curl -X POST https://api.firecrawl.dev/v2/scrape \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/login",
    "formats": ["markdown"],
    "actions": [
      {"type": "wait", "milliseconds": 1000},
      {"type": "click", "selector": "#login-button"},
      {"type": "write", "selector": "#email", "text": "..."}
    ]
  }'

Full docs: https://docs.firecrawl.dev

Don'ts

• Don't crawl a whole site without --limit — credits burn fast.
• Don't reach for Firecrawl when web_fetch works — wasteful.
• Don't commit the API key to git.
• Don't use Firecrawl for YouTube transcripts — use a dedicated transcription tool.