Fix #134: Feature request: AI agent tarpit#167
Open
JiwaniZakir wants to merge 3 commits intoBlessedRebuS:devfrom
Open
Fix #134: Feature request: AI agent tarpit#167JiwaniZakir wants to merge 3 commits intoBlessedRebuS:devfrom
JiwaniZakir wants to merge 3 commits intoBlessedRebuS:devfrom
Conversation
Introduces a configurable tarpit to slow down and serve random word-salad content to AI crawlers. Disabled by default; enable via tarpit.enabled in config.yaml or KRAWL_TARPIT_ENABLED env var. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #134
Adds an opt-in tarpit feature to slow down and inject noise into responses served to AI crawlers. A new
tarpitsection inconfig.yamlexposes two knobs —enabled(defaults tofalse) anddelay_seconds(defaults to5) — parsed inConfig.from_dictinsrc/config.pyintotarpit_enabledandtarpit_delay_secondsfields.In
src/routes/honeypot.py,trap_pagegains a conditionalasyncio.sleep(config.tarpit_delay_seconds)after the existing response delay, and_generate_pageappends a hidden<div class="link-box tarpit-text">block containing output from the new_tarpit_text()helper, which samples randomly from a 90-word_TARPIT_WORDSlist to produce 60-word nonsense strings intended to pollute LLM training corpora.Modified files:
config.yaml(newtarpitblock),src/config.py(Configdataclass fields +from_dictparsing),src/routes/honeypot.py(trap_page,_generate_page, new_tarpit_textand_TARPIT_WORDS).Verified manually by enabling
tarpit: enabled: truewithdelay_seconds: 3, hitting a trap URL, and confirming the added delay and the injected word-salad<div>in the rendered HTML; feature remains completely inert withenabled: false.This PR was created with AI assistance (Claude). The changes were reviewed by quality gates and a critic model before submission.