Tool detail

Minifetch Technical SEO API

Checks whether Minifetch's crawler may fetch a given URL per robots.txt, returning allow/deny and crawl-delay without following redirects.

Categories: SEO marketing, Website analysis

Use when

check if a URL blocks the Minifetch crawler before extracting it
read a site's crawl-delay before running an SEO audit
re-check robots.txt after editing it to bypass the 24h cache

Not for

not for extracting page content - use Minifetch's extract endpoints
not a general robots.txt parser for arbitrary user agents - checks Minifetch's UA only

Inputs

GET query params: url (string, required, target to check), fresh (boolean, optional - bypass 24h robots.txt cache)

Admitted input schema

{
  "type": "object",
  "properties": {
    "fresh": {
      "type": "boolean",
      "description": "Optional. If true, bypasses the 24h robots.txt cache and re-fetches robots.txt for this check. Use after editing your robots.txt or crawl-delay (or adding/removing a Minifetch block) to confirm the change took effect."
    },
    "url": {
      "type": "string",
      "minLength": 0,
      "maxLength": 4096,
      "description": "The target URL to check against robots.txt"
    }
  },
  "required": [
    "url"
  ],
  "additionalProperties": false
}

Endpoint

Method: GET
URL: https://minifetch.com/api/v1/x402/preflight/url-check

Payment

Protocol: x402
Listed price: 0.001 USDC per call on Base
Payment method: exact
Network: eip155:8453
Asset: 0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913

Listed terms are catalog metadata. The caller supplies inputs and calls the tool outside Onelayer. If payment is required, the caller uses a compatible wallet to pay.

Source

Directory: Coinbase x402 Bazaar
Record: GET https://minifetch.com/api/v1/x402/preflight/url-check
Retrieved: 2026-09-30T20:37:07.382Z
Catalog file SHA-256: 6837ccddf886f0007cf3727ea9d27faaa5047b51b4f425345c554ce1c271a524
Catalog built: 2026-09-30T22:31:58.034Z