A modern, fast, GUI-powered tool that generates sitemap.xml and robots.txt for both local website folders and online websites. Built with CustomTkinter, BeautifulSoup, and Python’s XML tools.
Scans your website folder and builds a clean, structured sitemap:
- Detects
.html/.htmfiles - Auto-generates URL paths
- Supports folder and file exclusions
- Adds priority + lastmod fields
Crawls any website (same domain only), fully automated:
- Respects maximum crawl depth
- Only collects HTML pages
- Smart URL normalization
- Prevents duplicate pages
Generates a clean robots.txt including:
- Allowed pages
- Disallowed folders
- Disallowed files
- Auto-inserts sitemap URL
Powered by CustomTkinter:
- Dark theme
- Modern controls
- Clean layout
- Fast, non-blocking thread handling
customtkinter
requests
beautifulsoup4
Python built-ins (no need to install):
os, xml.etree, tkinter, datetime, threading, urllib.parse
git clone https://github.com/YOUR-USERNAME/REPO-NAME.git
cd REPO-NAMEpip install -r requirements.txtpython main.pyThe GUI will open automatically.
-
Select your website folder
-
Enter your base URL
-
Optionally exclude folders / files
-
Generate → outputs:
sitemap.xmlrobots.txt
-
Enter website URL
-
Choose crawl depth
-
Generate → outputs:
sitemap.xmlrobots.txt
- Structured
- Human-readable
- Auto-prioritized
- User-agent rules
- Disallowed items
- Automatic sitemap link
- Fully safe to use for GitHub-hosted websites
- Works offline for local folders
- Works online with any accessible website
Feel free to open issues or PRs!
If this project helped you, please give it a star ⭐ on GitHub!