Robots.txt check

Fetch a site's robots.txt and answer: may this user-agent crawl this path? Send POST /api/robots-check with the required field url and pay $0.001 per call over x402 or MPP (there is no free tier). It returns a JSON object with allowed, matchedRule, userAgent, path and sitemaps.

Returns the matched rule and all declared sitemaps.

Category: Network & domains · Tags: robots crawling scraping compliance

TRY IN PLAYGROUND →

Parameters

Example request

curl -i -X POST https://agent402.tools/api/robots-check \
  -H "Content-Type: application/json" \
  -d '{"url":"https://agent402.tools/api/pricing","userAgent":"MyAgent"}'

Without payment this returns HTTP 402 Payment Required with the exact price for robots-check; any x402 v2 or MPP client pays it and retries.

Example response

{
  "allowed": true,
  "matchedRule": "Allow: /",
  "userAgent": "MyAgent",
  "path": "/api/pricing",
  "sitemaps": [
    "https://agent402.tools/sitemap.xml",
    "https://agent402.tools/sitemapindex.xml"
  ]
}

From an MCP client

catalog.call {
  "slug": "robots-check",
  "params": {
    "url": "https://agent402.tools/api/pricing",
    "userAgent": "MyAgent"
  }
}

The hosted connector at https://agent402.tools/mcp needs a payment for robots-check; the stdio package pays it from a wallet or from AGENT402_CREDITS_KEY. Local install: npx -y agent402-mcp.

Errors and behavior

• url is required. An input the tool rejects returns an HTTP 4xx whose body carries error, tool, expected, required and example, so the caller can correct it.
• A paid call that ends in any status of 400 or above is not charged over x402, MPP or a prepaid credits key: settlement is cancelled when the tool fails. The exception is a Tempo push credential, a transfer the buyer sent before the call: it settles before the tool runs, so if the tool then fails the payment is recorded as a refund owed to the paying wallet.
• Wallet-only: this tool reaches the network or stored state, so it has no proof-of-work tier. A prepaid card-credits key issued earlier (Authorization: Bearer a402_...) also pays it.
• A GET or HEAD to /api/robots-check returns the same 402 quote, so the price can be read without a body.
• An Idempotency-Key header makes a retried paid call replay the first 200 instead of charging again (an answer larger than 1 MB is not replayed).

Paid call (JavaScript agent)

import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client();
client.setSpendControls?.(false); // keep your own spending ceiling in code
registerExactEvmScheme(client, { signer: privateKeyToAccount(KEY) });
const payFetch = wrapFetchWithPayment(fetch, client);

const res = await payFetch("https://agent402.tools/api/robots-check", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "url": "https://agent402.tools/api/pricing",
    "userAgent": "MyAgent"
  }),
});

Part of these workflows

Robots.txt check is one step in these 5 skill packs, each sold as a single call:

• DNS & network ops - End-to-end DNS health check: records, multi-resolver propagation, WHOIS, ASN, robots.txt, and reachability.
• Site status snapshot - The 'is this site healthy, addressable, and crawlable - right now?' workflow. DNS resolution → HTTP reachability → security headers → TLS certificate expiry → robots policy. Five tools, one structured status payload an operator (or an uptime bot, or a pre-flight check before an extract/crawl) can act on.
• Technical SEO audit - Can search engines and AI crawlers actually index this page? One pass over reachability, TLS, robots policy, sitemap health, meta/OpenGraph tags, and the on-page link graph.
• Domain intel - Full domain security and SEO intelligence in one call: WHOIS, DNS, TLS cert, HTTP headers, tech stack, robots policy, and certificate transparency.
• Page audit - Full page SEO + security audit: content extraction, metadata, HTTP headers, robots policy, and sitemap health in one call.

Related tools