CrawlPay SDK

Charge AI bots $0.001 USDC per page. Two lines of code.

GPTBot, ClaudeBot, PerplexityBot crawl your site constantly. They read your content, train models on it, and pay you nothing. CrawlPay fixes that - bots that handle HTTP correctly pay via x402, the rest pass through untouched.

Install

npm install @crawlpay/sdk

Quick start

Next.js

import { crawlpay } from "@crawlpay/sdk"
import { NextResponse } from "next/server"

const paywall = crawlpay({ wallet: "0xYourWallet" })

export function middleware(request) {
  return paywall(request) ?? NextResponse.next()
}

Express

import { crawlpayExpress } from "@crawlpay/sdk/express"
import express from "express"

const app = express()
app.use(crawlpayExpress({ wallet: "0xYourWallet" }))

Cloudflare Workers

import { crawlpayCloudflare } from "@crawlpay/sdk/cloudflare"

const paywall = crawlpayCloudflare({ wallet: "0xYourWallet" })

export default {
  async fetch(request, env, ctx) {
    // Returns 402 for unpaid bots; forwards humans and paid bots to origin
    return paywall.fetch(request)
  }
}

For manual control (you handle forwarding yourself):

import { crawlpayCheck } from "@crawlpay/sdk/cloudflare"

const check = crawlpayCheck({ wallet: "0xYourWallet" })

export default {
  async fetch(request) {
    const blocked = check(request)
    if (blocked) return blocked
    return fetch(request)
  }
}

Vault Mode (Story CDR)

Private encrypted datasets - content that doesn't exist in plaintext anywhere. Bots pay, Story Protocol decrypts.

crawlpay({
  wallet: "0xYourWallet",
  vault: process.env.CDR_VAULT_UUID
})

Configuration

Option	Required	Default	Description
wallet	Yes	-	Your Arc wallet address
price	No	"0.001"	Default price in USDC
paths	No	-	Per-path prices via patterns
network	No	"base"	Payment network (base or polygon)
vault	No	-	Story CDR vault UUID

Per-path pricing

crawlpay({
  wallet: "0xYourWallet",
  price: "0.001", // default for all pages
  paths: {
    "/premium/*": "0.01", // 10x for premium content
    "/api/*": "0.005", // 5x for API endpoints
    "/vault/*": "0.001", // CDR vault pages
  },
})

Supported AI bots

Bot	Provider
GPTBot	OpenAI
ChatGPT-User	OpenAI
ClaudeBot	Anthropic
anthropic-ai	Anthropic
GoogleOther	Google
Google-Extended	Google
PerplexityBot	Perplexity
CCBot	Common Crawl
Bytespider	ByteDance
FacebookBot	Meta
Applebot-Extended	Apple

Links

• Demo: crawl-pay.com
• Dashboard: crawl-pay.com/dashboard
• GitHub: divergenttt/CrawlPay-Vault-SDK
• App: divergenttt/CrawlPay-Vault
• License: MIT