Dedupe lines

Remove duplicate lines, preserving first-seen order. Send POST /api/dedupe-lines with the required field text and pay $0.001 per call over x402 or MPP, or call it free by solving a proof-of-work challenge. It returns a JSON object with result, removed and kept.

Returns the deduped text and how many were removed.

Category: Text processing · Tags: dedupe unique lines text

TRY IN PLAYGROUND →

Parameters

Example request

curl -i -X POST https://agent402.tools/api/dedupe-lines \
  -H "Content-Type: application/json" \
  -d '{"text":"a\nb\na\nc\nb"}'

Without payment this returns HTTP 402 Payment Required with the exact price for dedupe-lines; any x402 v2 or MPP client pays it and retries.

Example response

{
  "result": "a\nb\nc",
  "removed": 2,
  "kept": 3
}

From an MCP client

catalog.call {
  "slug": "dedupe-lines",
  "params": {
    "text": "a\nb\na\nc\nb"
  }
}

On the hosted connector at https://agent402.tools/mcp, catalog.call runs dedupe-lines free (rate-limited, no wallet). Local install: npx -y agent402-mcp.

Errors and behavior

• text is required. An input the tool rejects returns an HTTP 4xx whose body carries error, tool, expected, required and example, so the caller can correct it.
• A paid call that ends in any status of 400 or above is not charged over x402, MPP or a prepaid credits key: settlement is cancelled when the tool fails. The exception is a Tempo push credential, a transfer the buyer sent before the call: it settles before the tool runs, so if the tool then fails the payment is recorded as a refund owed to the paying wallet.
• Free tier: no outbound network call leaves the server for this tool, so proof-of-work (16 leading zero bits of sha256) pays for it.
• A GET or HEAD to /api/dedupe-lines returns the same 402 quote, so the price can be read without a body.
• An Idempotency-Key header makes a retried paid call replay the first 200 instead of charging again (an answer larger than 1 MB is not replayed).

Paid call (JavaScript agent)

import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client();
client.setSpendControls?.(false); // keep your own spending ceiling in code
registerExactEvmScheme(client, { signer: privateKeyToAccount(KEY) });
const payFetch = wrapFetchWithPayment(fetch, client);

const res = await payFetch("https://agent402.tools/api/dedupe-lines", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "text": "a\nb\na\nc\nb"
  }),
});

No wallet? Pay with compute

Fetch a challenge, solve the sha256 puzzle (16 leading zero bits, a fraction of a second of CPU), and resend with the X-Pow-Solution header:

import { createHash } from "node:crypto";
const lz = (b) => { let t = 0; for (const x of b) { if (!x) { t += 8; continue; } t += Math.clz32(x) - 24; break; } return t; };
const c = await (await fetch("https://agent402.tools/api/pow/challenge?slug=dedupe-lines")).json();
let n = 0;
while (lz(createHash("sha256").update(c.challenge + ":" + n).digest()) < c.difficulty) n++;
await fetch("https://agent402.tools/api/dedupe-lines", { method: "POST", headers: { "X-Pow-Solution": c.token + ":" + n, "Content-Type": "application/json" }, body: JSON.stringify({"text":"a\nb\na\nc\nb"}) });

Part of these workflows

Dedupe lines is one step in this skill pack, each sold as a single call:

• Text hygiene - Turn a wall of dirty text - chat logs, scraped pages, user-generated content, log dumps - into something safe to store, search, and pipe into the next step. Measure first, redact PII before anything else touches the data, then dedupe, sort, extract entities, surface keywords, and grade the readability of what's left.

Related tools